1. Overview of Phase 4 Performance Validation Methods and Techniques

2. Objectives in Phase 4 Validate Performance –Prove your solution meets the design requirements. –More specifically, measure performance and show your key metrics/target have been achieved. Transition project to the Sponsor’s Business –Identify key next steps –Documentation (Manuals, Training, Safety)

3. Performance Validation 1.Determine what you need to quantify 2.Develop test plans and a testing program 3.Execute tests and interpret results 4.Connect results to design requirements 5.Analyze failures 6.Modify? Redesign? Terminate Project? 7.Repeat all or part of the testing program. 8.Communicate results.

4. Performance Validation... Determine what you need to quantify -Direct or indirect measurement of a metric -How certain do you need to be regarding a particular Target Value? Accuracy? Repeatability? Reliability? –Other dimensions of performance Production costs for manufacturing Value?

5. Quick Example Automatic filling a vial with liquid Target value is 90% fill Must be at least 85% filled Measurements from 12 tests: 92 91 93 85 86 93 88 95 87 89 87 95 Mean = 90.1 90% If the goal is 95% certainty, then the expected range of fill would be +/- two standard deviations about the mean which translates to a range of 83.1 to 97.1... St.Dev. = 3.50

6. Test Plan... and Beyond See the SD web site for more details Key Message –a good plan can be executed by anyone –collectively, a set of test plans (i.e. a testing program) along with appropriate analysis and interpretation will produce a convincing argument that your teams' designs are clearly aligned with and meet your customers’ needs.

7. Failure Analysis Every product or process has modes of failure. An analysis of potential failures helps designers focus on and understand the impact of potential process or product risks and failures. Several systematic methodologies have been develop to quantify the effects and impacts of failures.

8. Failure Modes and Effects Analysis (FMEA) Example: Hydraulic Hose Failure Severity, Occurrence and Detection ratings are based on a 1 = low to 10 = high scale. The FMEA results clearly show the greatest risk is associated with over- pressure failure, and the lowest risk is due to weathering-related failure. The excel template shown is above is available at: http://research.me.udel.edu/~jglancey/FMEA_Template.xls

9. Performance Validation 1.Determine what you need to quantify 2.Develop test plans and a testing program 3.Execute tests and interpret results 4.Connect results to design requirements 5.Analyze failures 6.Modify? Redesign? Terminate Project? 7.Repeat all or part of the testing program. 8.Communicate results.

10. Questions?

11. A Test Plan... -Objective of the Test -Description of Test Rationale, and Procedures -Experimental Plan -Measured or Response Variable (s) -Variables to Changed (Factors) -Variables to Held Constant -Procedures to prepare test samples or materials -Data Collection Plan -Data Analysis Methods -Risks and Contingencies -Failure Analysis (when appropriate) -Resources –Sensors –DAQ Hardware, Instruments –Test Equipment and Hardware –People (Who, What, When)

12. Testing Procedure A set of test plans Collectively, results from executing the testing procedure, properly analyzed, should provide the proof that your concept has met the design requirements. The implication is that creating a prototype is not the end goal!