Presentation on theme: "Langley Research Center Applications of the Modern Design of Experiments at NASA Langley Research Center Richard DeLoach NASA Langley Research Center 46."— Presentation transcript:
Langley Research Center Applications of the Modern Design of Experiments at NASA Langley Research Center Richard DeLoach NASA Langley Research Center 46 th Annual Fall Technical Conference Learning and Discovery - Building the Future Through Quality and Statistics October 17-18, 2002 Valley Forge, PA
Langley Research Center OUTLINE What is the “Modern” Design of Experiments (MDOE)? Conventional wind tunnel test practices –Brief description –Some shortcomings Reasons for Langley’s growing interest in statistically designed experiments –Resource savings –Enhanced quality and productivity –Deeper insights into underlying physics of flight Summary and parting thoughts
Langley Research Center Why the “M” in MDOE? (How does Modern DOE differ from “ordinary” DOE?) Naming convention to distinguish between modern and conventional experiment design methods. –LaRC researchers “design experiments” all the time –Emphasis on modern experiment design overcomes initial perception that this is nothing new. Effort to highlight differences between industrial and scientific applications of DOE –Some unsatisfactory prior experiences with industrial DOE –Perceptions that designed experiments inappropriate for complex research applications
Langley Research Center Four Eras in the History of Designed Experiments (Credit: Douglas Montgomery) The agricultural origins, 1918 – 1940s –R. A. Fisher & his co-workers –Profound impact on agricultural science –Factorial designs, ANOVA The first industrial era, 1951 – late 1970s –George Box and others –Response surface methods –Applications in the chemical & process industries The second industrial era, late 1970s – 1990 –Quality improvement initiatives in many companies –Taguchi and robust parameter design, process robustness The modern era, beginning circa 1990
Langley Research Center Conventional Wind Tunnel Testing Characteristics of a typical wind tunnel test Design –Test matrix changes One Factor At a Time (OFAT). –Test matrix generally specifies greatest combination of factor levels that resources permit. Execution –Change variable levels monotonically to maximize data acquisition rate. –Make small incremental changes in each factor. –Physically set all factor combinations of interest. –Data collection ends when resources are exhausted. Analysis –Tabulate measured responses (generate database). –Graphically display selected response relationships.
Langley Research Center Three Shortcomings in OFAT Testing MDOE is not simply an incremental improvement in an otherwise satisfactory process. Design –Changing one factor at a time is very time-consuming. –Designing for maximum data volume also ensures maximum time, labor, and expense. Execution –Monotonically changing factor levels confounds true factor effects with ubiquitous systematic variations. –Fastest test matrix not the highest quality test matrix. Analysis –There are generally more factor effects of interest than can be displayed efficiently. –Important relationships, especially factor interactions, are often not quantified.
Langley Research Center Two Competing Views of Wind Tunnel Testing (“Knowledge vs. Data”) OFAT: An industrial process –Model: Wind tunnel as data factory –Product: High quality data –Goal: Maximize data volume, given fixed tunnel time. MDOE: A research process –Model: Wind tunnel as laboratory –Product: Insights leading to reliable response prediction –Goal: Minimize expense and time to achieve specific technical objectives
Langley Research Center Importance of Cycle Time Reduction in Aerospace Testing Dollars are often perceived by researchers as the most important resource, but… “Cycle Time” can be more critical. –The cost of bringing a major aerospace product to market can be in the billions of dollars. –Cost of capital can therefore be in the millions of dollars per day. –Cost of getting to the market late can be very high.
Langley Research Center OFAT Thinking Maximizes Cycle Time Emphasis on direct measurement of all factor combinations results in high-volume data collection. Seldom sufficient time to set all conditions of interest. Productivity associated with high data volume, but… Data acquisition rate is limited so “high productivity” translates into tests that use all available time. The “Bernoulli Factor”
Langley Research Center High Data Volume – A Very Old Idea! Even the most stupid of men, by some instinct of nature, by himself and without any instruction, is convinced that the more observations that have been made, the less danger there is of wandering from one’s goal. Jacob Bernoulli ( )
Langley Research Center Synthetic Jet Lift Enhancement Study An illustration of why no amount of time is “too much” for an OFAT wind tunnel test.
Langley Research Center Leading Edge Synthetic Jets Enhanced Lift Without Synthetic Jet Flow Control Lift With Synthetic Jet Flow Control
Langley Research Center Starboard Flaps Outboard (Mean + Diff) Inboard Starboard Jet Banks SJ4 SJ3 SJ2 SJ1 Port Jet Banks PJ1 PJ2 PJ3 PJ4 Port Flaps Inboard Outboard (Mean + Diff) Synthetic Jet Configuration Variables Four jet variables, 3 flap variables each side
Langley Research Center Synthetic Jet Data Volume Considerations There are 14 configuration variables –Four banks of synthetic jets each wing –Three flap variables each wing Inboard flap deflection Outboard flap deflection Differential outboard flap deflection (clamshell mode) At least three levels of each factor required for any response that is non-linear in the factors. Total number of candidate configurations: 3 14 =4,782,969.
Langley Research Center Synthetic Jet Data Volume Considerations Total number of candidate configurations: 3 14 =4,782,969. Required time to measure all configurations assuming two configurations/minute, two shifts/day: 9.6 years Allocated test time: two weeks –There is time to set a theoretical maximum of 19,200 configurations –This would leave 4,763,769 configurations unexamined (99.6% of the cases). In practice, many fewer configurations could be set. Any realistic allocation of wind tunnel time is likely to be considered insufficient. Conventional solution is to rely on judgment and “instinct” to select some subset of configurations to study, but… –This assumes we already know much of the answer. –Much must go unexamined in any case.
Langley Research Center Minimum Number of Candidate Configurations (two levels per factor) Grows exponentially with number of factors 14 16,384
Langley Research Center What Should Determine Data Volume? The MDOE approach to developing a test matrix. Not practical to set every factor combination, even with a moderate number of factors. How can we rationally decide how many and which factor combinations to set? –Number of factor combinations driven by signal/noise ratio, inference error risk tolerance, and expected complexity of response. –Selection of factor combinations driven by specific experiment design.
Langley Research Center Factors That Drive Data Volume Requirements Size of smallest effect that must be resolved, Variance in individual measurements, 2 Acceptable inference error risk –Probability of inferring the wrong size of an effect, –Probability of failing to detect a significant effect, Parameters in model, p Note Bene: There are variations on this formula, depending on the details of the experiment, but they all feature the same elements: S/N, risk, model complexity
Langley Research Center Cost/Benefit Tradeoff Synthetic Jet Lift Enhancement Study = 0.01 = p = 20 = %95% 92
Langley Research Center MDOE Data Volume Estimate Synthetic Jet Lift Enhancement Study We estimated that 92 or more factor combinations would satisfy our inference error risk tolerance. –For the smallest signal to noise ratio expected –For the most complex response model expected Sufficient volume of data to ensure 99% probability of detecting lift changes large enough to be important to us. Sufficient volume of data to ensure 95% probability that indicated lift change is within an acceptably small range. Selection of factor combinations driven by design.
Langley Research Center Impact of Experiment Design on Data Volume Synthetic Jet Lift Enhancement Study A two-level factorial design would reveal flap/jet interaction effects, which are of special interest, but… A full factorial design in 14 factors would require more than the two weeks allotted. Fractional factorial design a possible solution –Confounds low- and high-order interactions (aliasing). –Good solution if high-order interactions are negligible. –Unfortunately, high-order interactions were anticipated. Solution was to exploit expected independence of port- side and starboard-side factor effects. This reduces design to two additive full factorial designs in seven factors each.
Langley Research Center Minimum Number of Candidate Configurations (two levels per factor) Grows exponentially with number of factors ,384
Langley Research Center Synthetic Jet Lift Enhancement Study Initial MDOE Experiments Minimum of 92 factor combinations to satisfy inference error risk tolerance. Two full factorial designs (port wing and starboard wing) in seven factors –2 7 = 128 factor combinations each –Sufficient to drive inference error risk below acceptable levels. All main effects and interactions through 7-way on each wing, clear of any aliasing. Assumes negligible port/starboard interactions –In harmony with views of aerodynamic subject-matter specialists –Assumption easy to test Less than 5 wind-on hours.
Langley Research Center Summary of Single-Wing Results There were over a dozen lift effects significant at the 0.05 level (95% confident they are real) Flap effects dominated, as expected, however… There were significant jet main effects. There were significant interactions up to 6-way. There were substantial jet/flap interactions. Some asymmetry between wings noted in that port and starboard lift effects were not identical. –Some airframe manufacturing differences –Some differences in jet performance
Langley Research Center Synthetic Jet Lift Enhancement Study Follow-On MDOE Experiments Initial port-side and starboard-side experiments were extended to include angle of attack and replicated for two synthetic jet waveforms. These extended experiments were augmented to permit higher-order response modeling. –2 nd -order terms in jet/flap configuration variables –Up to 4 th -order in angle of attack Model believed capable of adequately predicting lift for all factor combinations in range tested. Total wind-on time for all experiments: 22 hrs (3 shifts) Approximately 25% of wall-clock time is wind on, so… –This spanned roughly 11 eight-hour shifts –Less than six two-shift days (60% of allocated tunnel time)
Langley Research Center Current Status of MDOE Over 40 MDOE tests designed, executed, and analyzed since concept introduced to Langley production wind tunnels in Over 100 researchers have been intimately exposed to these methods through courses and/or scientific collaboration. Employees at Langley and from private industry have taken graduate leave to obtain university training in formal experiment design. The American Institute of Aeronautics and Astronautics (AIAA) now includes MDOE instruction in its Professional Development Series of short courses. MDOE recognized as an emerging technology in two AIAA Recommended Practices documents (wind tunnel testing and force balances) Technical sessions on experiment design now a part of major AIAA conferences (Aerospace Sciences, Ground Testing, Applied Aeronautics)
Langley Research Center A Parting Thought Based on balance of trade data, the world especially values two American products (Credit: Dennis Bushnell, LaRC Chief Scientist) –Aeronautical systems –Wheat The one we most efficiently cultivate is wheat. There is an important role for the statistics and quality community in aeronautical research, because... Statistically designed experiments are key to cultivating aeronautics as efficiently and productively as wheat.