A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment Atlee M. Cunningham, Jr., Jon Hagar, Ryan J. Holman Lockheed Martin

2 Agenda  Introduce the trade study space: the F16 and Combinatorial Test (CT) problem  Define the F16 Failure  Present the steps of the CT study  Cover the results  Conclusions

3 I NTRODUCTION : F16 V ENTRAL F IN S TUDY AND A PPLYING C OMBINATORIAL T ESTING  Evaluate the use of Combinatorial Testing (CT) to a real “problem” – Used a historic F-16 problem and data – See if CT could be used in place of or to support an expert Save time and/or people Mixed example, between what was done and what could have been done  Problem space – Interacting factors (good for CT) – Outside of the software testing – System-Hardware failure resolution and design evaluation – Demonstrate CT is viable

4 F16 Failure Case Study  During production and maintenance of the F-16 fighter aircraft a structural problem immerged – Buffeting of the F-16 ventral fins has provided a classic example of structural fatigue of such aerodynamic surfaces by an upstream source of severe turbulent wakes – These fins are very thin surfaces, about 5 ft. chord and 2 ft. span, composed of three wedge like surfaces that taper down to edge thicknesses of 0.05 inches, all of which makes the fins susceptible to turbulence buffeting – Examples of possibly interacting turbulence sources include: various centerline stores, side slips and inlet lip spillage during rapid decelerations  The historic work done by Atlee M. Cunningham, Jr. and Ryan J. Holman  The Combinatorial analysis and case study was primarily done Jon Hagar with support from Atlee Cunningham as the “expert”

5 O RIGINAL F-16 P ROBLEM D ETAILS  Added 2 avionics LANTIRN pods on the F-16 just aft of the inlet on the lower fuselage directly upstream of the ventral fins  Avionics pods in general are often not very aerodynamic in shape and hence can produce very turbulent wakes  The damage to the right hand ventral fin on first flight with LANTIRNS – Originally, the primary source of the fin’s fatigue and loss was high speed throttle chops that produced severe turbulence from inlet lip spillage during rapid decelerations where the throttle was suddenly pushed to idle position  A comparison of the ventral fin response to LANTIRN and throttle chop turbulence was done – The response levels are about the same; however, constant buffeting by the LANTIRNS produced much higher fatigue damage per flight hour as compared to that due to the transient throttle chop – As a result, several major structural re-designs of the fins and other associated structures followed over the following years that incrementally improved the fatigue life of these components

6 F16 with LANTIRN Pod and Ventral Fin

7 Failure: Damaged Ventral Fin But Why (what parameters and interactions)?

8 O RIGINAL A NALYSIS H ISTORY  After a number of years more the problem continued.  As a result, a detailed analysis of the flight data was performed by Atlee Cunningham, yielding –Showed that the most severe buffeting of the ventrals consistently occurred with only LANTIRN pods on the aircraft and with high speed throttle chops at Mach 0.95 on the clean aircraft –Anomalous trends were also seen in throttle chop data with LANTIRNs where response levels were 3-to-4 times as high as level flight with LANTIRN –Recognizing that the very thin ventrals (leading edge thickness is 0.05 in.) would probably be subject to leading edge separation at small angles of side slip –Flow change resulted in a large increase in the slope of side force with side slip angle, which would have a significant impact on dynamic loads due to large amplitude turbulence

9 Do More Testing and Analysis (by experts)  A low speed small scale wind tunnel test was conducted to –Explore various aspects of ventral aerodynamics and effects of modifications –Data were obtained with 1/5 scale models of the fin mounted on the wind tunnel wall and rotated for incidence effects –Testing was to determine “sensitivities” (variables and values) but had to be designed by expert  Flight Tests were conducted –Three of these four fins, plus several early block ventral fins, were tested on an early Block 15 F-16. The fins consisted of: (1) the baseline fin, “BSLN,” the standard Block 40 ventral fin; (2) the “MMC” fin, the Block 40 fin with 40% stiffer skins of MMC aluminum material (3) the “MMCNC” fin, the MMC fin with an added rounded “nose cap” glove with a NACA 0012 airfoil section of 5 inch chord (4) the “NACA” fin, the Block 40 modified to have a full span, full chord airfoil section that eliminated the sharp leading edge and sharp tip section of the fin –An expert had to define test program (combinations) for these too = hundreds of hours

10 D EFINING CONDITIONS FOR CT  What was the situation(s) that brought the failure on? Factors considered include: –Aircraft (AC) –Maneuver –Speed (Mach) –Altitude –Aircraft add on structures (tanks, pods, etc.)  Which design solutions (4 fin designs) might solve the problem with different aircraft configurations: –Block 15 –Block 40

11 I DEALIZED A NALYSIS S TEPS USING CT (H YPOTHETICAL R ECONSTRUCTION )  NIST ACTS Combinatorial Tool was used to “reverse” engineer a test program –Other tools were considered –Open source nature was deciding factor  This can be viewed as a “reverse” or “Re” engineering case study –We were trying to see if the tool would replicate the historic test program without an system expert –Test planning using a CT tool (not the expert )  A series of idealized steps were done using the tool

12 CT Step 1  Historic “first” test program - clean baseline configuration, which in the example are F16s in block 15 and 40 in “clean” configuration, and apply “testing” to points associated  Input to tool (equivalence classes): Tool produced: 90 test cases (similar to actual effort) with 2 way Aircraft15, 40 Altitude (s)5k, 10k, 15k, 20k, 30k, 40k, 50k Maneuvershi-speed throttle, slow accel/dwell, L/R 5deg side slip, L/R 360 roll, R/L 360 roll, R/L 5deg side slip, Med accel/dwell, R-L- R-L banking, Hi-speed to Low, 360 nose roll Mach(100 th )40, 50, 60, 70, 80, 90, 100, 110, 120 Parameters: Variable:

13 Step 2: Refinement of the test program  Step 2 a more refined set of analyses would have been done based on information from: –Step 1 –Historic databases –More detailed wind tunnel analyses –Supplemental water tunnel analysis –Flight data and constraints were available  This effort confirmed design work –Produced 30 test cases from 2 way coverage Parameters: Variable: Aircraft15, 40 Alt5, 10, 15 Mach(100 th )60, 80, 85, 90, 95 LANTIRNon, off

14 Step 3: Final Design Flight Test Program Parameters: Variable: AC-BLK&Ventral-Fin-Config Blk15-Blk15 ventral, Blk15-Blk40 ventral, Blk15-MMC ventral, Blk15- MMC ventral + cap, Blk40-Blk40 ventral, Blk40-MMC ventral, Blk40-MMC Ventral+ nonosecap, Blk40-NACA LANTIRN [on, off] Alt [5, 10, 15] Mach (100ths) [60, 70, 80, 85, 90, 95] Maneuver Block [basic, basic +]  Tests for a flight test program  Number of test (cases) generated with 2 way coverage: 72

15 Conclusions  An example where combinatorial test could have aided – Provided another test design method for teams to use – Reduce the "shotgun” approach and expert judgment needed for situations dealing with “many” parameters –Showed CT can support a system failure (fault isolation) analysis  Historic data useful in a CT proof on concept (case study example)  Lockheed Martin will continue advocating CT as a technique –Looking for pilots and more data points –Would be interesting to compare to DOE approaches –How to get Engineers to start using  Other items noticed –Tool interchange (operability), particularly into a test automation framework –Constraints were “tricky” –Interface to/from Model based testing would be useful

16 Summary  Demonstrated Combinatorial Test tool could have supported the F16 problem (or other hardware, software, system test/analysis) –Expert felt results would have been similar –Approach could support other programs  Open source tool worked –Commercial tools worked too –Supports move from “theory” to real use  Supported an “non” software area – System Design/Failure Evaluation