1 An Overview of MSFC Quantitative Risk Assessment (QRA) Practices Fayssal Safie/MSFC October 25, 2000

2 Agenda Quantitative Risk Assessment System (QRAS) Other PRA-Related Practices Reliability Prediction Probabilistic Structural Analysis Similarity Analysis Reliability Demonstration

3 MSFC Propulsion Elements QRAS

4 QRAS Background Space Shuttle probabilistic risk assessment (PRA) studies Space Shuttle PRA for Galileo mission (PRC) Galileo PRA update (SAIC) Space Shuttle PRA (SAIC) 1997/ Space Shuttle PRA (NASA/Code Q)

5 QRAS Background (cont’d) 1997/2000 NASA QRA study In July 1996, the NASA Administrator directed NASA Headquarters to develop a software system to quantitatively assess the overall shuttle risk and serve as a tool to estimate risk changes due to proposed shuttle upgrades. At the request of NASA Headquarters, MSFC and JSC, supported by their prime contractors, are modeling their respective elements. The software system, called QRAS (Quantitative Risk Assessment System), is designed and developed by NASA Headquarters Code Q.

6 Develop a quantitative risk model to: Assess the reliability/risk of the overall shuttle vehicle, its major elements, and their components Evaluate risk reduction due to proposed shuttle upgrades Rank shuttle failure modes Perform trade studies/sensitivity analyses QRAS Objectives

7 QRAS Model Requirements Model builds on work done by SAIC Shuttle PRA model. Model is modular, reflecting shuttle modularity with its discrete elements, subsystems, and components (flexible to accommodate upgraded components and additional details). Model must be most detailed in high risk areas to allow sensitivity analysis and trade studies to be performed. Model/tool must be user-friendly and easily updateable. Model must be capable of identifying, quantifying, and prioritizing the major risk contributors. Model must support NASA decision-making process (evaluating shuttle upgrades and supporting flight issues).

8 QRAS Modeling Approach Space Shuttle ORBITER MCC HEX HPFTPLPFTP Products 1. Space Shuttle Risk 2. Element Risk 3. Subsystem Risk 4. Risk Ranking 5. Sensitivity Analysis etc.. Turbine Blade Porosity Turn- Around Duct Fail. Housing Retaining Lug Fail. FLIGHT/TEST DATA PROBABILISTIC STRUCTURAL MODELS SIMILARITY ANALYSIS ENGINEERING JUDGMENT System Hierarchy Turbine Blade Porosity Mission Success Inspection Not Effective Porosity Present in Critical Location ET SSME ISRB UNCERTAINTY DISTRIBUTION FOR LOV DUE TO TURBINE BLADE POROSITY Event Tree RISK AGGREGATION OF BASIC EVENTS Functional Event Sequence Diagram (FESD) End State or Transfer Porosity Present in Critical Location Leads to Crack in <4300 sec Scenario Number 1LOV 3 MS 4 2 Turbine Blade Porosity Inspection Not Effective Porosity Present in Critical Location QUANTIFICATION OF FESD INITIATING & PIVOTAL EVENTS UNCERTAINTY DISTRIBUTION FOR EVENT PROBABILITY EVENT PROBABILITY DISTRIBUTION BASIC/INITIATING EVENTS Porosity in Critical Location Leads to Crack in <4300 sec Mission Success Mission Success Loss of Vehicle (LOV) Blade Failure Mission Success Blade Failure 5MS

9 QRAS MSFC Team Participants MSFC Safety & Mission Assurance (S&MA) Chief engineer & project offices Engineering Prime contractors Reliability engineering Design & manufacturing engineering Hernandez Engineering Inc. (HEI) Reliability engineering and simulation

10 QRAS Databases Problem Reporting and Corrective Action (PRACA) Automated Configuration data Tracking System (ACTS) Logbooks Engineering data/analyses Generic data Lessons learned SAIC study

11 QRAS Propulsion Element Models Significant Observations Strength: QRAS modeling effort has contributed towards drawing management attention in using statistical and probabilistic information in the decision making process. Event Sequence Diagram (ESD) provides a better understanding of the failure mode risk and an excellent way to address risk mitigation. Data contained in the individual ESD packages are an excellent source of reference material and lessons learned. QRAS models constitute: The best source of failure rate data for the shuttle program to evaluate upgrades. The best source of information to understand the risk mitigation in place. The best source to understand the physics of failure for critical failure modes/events.

12 Considerations: QRAS is a large scale QRA study which is very complex and require extensive knowledge of the system, a large amount of data, and extensive modeling. Use of engineering judgment introduces significant amount of uncertainty. Quantification methods, in most cases, are not robust. Overlooking one piece of data may dramatically change the probability of loss of vehicle. QRAS Propulsion Element Models Significant Observations (cont’d)

13 Considerations (cont’d): Modeling of human error/process error is a big challenge. Human error/process error has been incorporated implicitly where flight and test data exist. For structural failures which are modeled using design information, the human error/process error has been incorporated explicitly using placeholders based on historical data. The QRAS modeling effort has shown that developing explicit models for the human error/process error is extremely difficult because of lack of adequate data. QRAS Propulsion Element Models Significant Observations (cont’d)

14 Considerations (cont’d): QRAS/PRA failure probabilities are imbalanced Some failure probabilities are derived using mainly design information ( P&W Turbopumps), while others are derived using mainly test and flight data (RKDN SSME hardware). Generic data are also used in other cases. Some failure probabilities are derived using limited data (solid propulsion elements), while others are derived based on a lot of data (liquid propulsion elements). Difficult to model common cause failures Incomplete interface models QRAS Propulsion Element Models Significant Observations (cont’d)

15 QRAS Conclusions Following a well defined and documented systematic procedure, involving the appropriate disciplines (reliability, design, and manufacturing engineering), and using the appropriate data are the key elements for a successful QRA study. Information derived from QRA studies are most accurate and useful at lower levels (within components and failure modes). QRAS tool is the best QRA tool available to support the shuttle program management decisions.

16 Other PRA-Related Practices Reliability Prediction

17 Reliability Prediction Reliability prediction techniques are dependent on the degree of the design definition and the availability of historical data. Two commonly used techniques are: Probabilistic design techniques: Reliability is predicted using engineering failure models. Similarity analysis techniques: Reliability of a new design is predicted using reliability of similar parts.

18 Reliability Prediction Probabilistic Structural Analysis It is a tool to probabilistically characterize the design and analyze its reliability using engineering failure models. It is a tool to evaluate the expected reliability of a part given the structural capability and the expected operating environment. It is used when failure data is not available and the design is characterized by complex geometry or is sensitive to loads, material properties, and environments.

19 FRACTURE LOCATION During rig testing the AT/HPFTP Bearing experienced several cracked races. Summary of 440C race fractures / tests: 3 of 4 Fractured Reliability Prediction Probabilistic Structural Analysis (cont’d) Turbo-Pump Bearing Example

20 OBJECTIVE: Predict probability of inner race over-stress, under the conditions experienced in the test rig, and estimate the effect of manufacturing stresses on the fracture probability. Stress Allowable Load Failure Region Reliability Prediction Probabilistic Structural Analysis (cont’d) Turbo-Pump Bearing Example

21 Conditions Using rig fits and clearances Crack size data from actual cut-ups Stresses associated with manufacturing (ideal) Materials properties and their variations Failure mode being analyzed is over-stress Reliability Prediction Probabilistic Structural Analysis (cont’d) Turbo-Pump Bearing Example

22 HPFTP Roller Bearing Inner Race - Model Flow Randomly select values for inner race material properties  Randomly select values for shaft and sleeve material properties  Tolerance fits of rig test bearing Inner race hoop stress contribution at given conditions Shaft and sleeve hoop stress contribution at given conditions. Total hoop stress Stress due to Manufacturing Stress > Allowable Load Iterate and compute Failure Probability Variation in: o Fracture Toughness o Yield Strength o No. of Cracks o Crack Depth o Crack Length Compute Allowable Load for each crack Compute Allowable Load (worst crack) Reliability Prediction Probabilistic Structural Analysis (cont’d) Turbo-Pump Bearing Example

23 RESULTS - FAILURE RATES At Test 3 of 4 failed --- In 15+ tests never had a through ring fracture Race Configuration 440C w/ actual manufacturing stresses (ie ideal + abusive grinding) 440C w/no manf. stresses 440C w/ideal manf. stresses 9310 w/ ideal manf stresses Probabilistic Structural Analysis 68,000 fail/100k firings 1,500 fail/100k firings 27,000 fail/100k firings 10 fail/100k firings It is estimated that 50% of the through ring fractures would result in an engine shutdown. The shutdown 9310 HPFTP Roller Bearing Inner Race Failure Rate is then: 0.50 X 10/100k = 5 fail/100k firings Reliability Prediction Probabilistic Structural Analysis (cont’d) Turbo-Pump Bearing Example

24 Reliability Prediction Similarity Analysis Similarity Analysis is a technique for predicting the reliability of a new design based on historical data of similar designs (heritage hardware). Failure rates derived from historical data are modified to reflect the design and environment of the new hardware. Similarity Analysis is best performed at the lowest level possible, where more data is available and more appropriate judgment is made.

25 Reliability Prediction Similarity Analysis (cont’d) Fuel Turbo Pump Example Assume a Fuel Turbo Pump (FTP) has a historical failure rate of: 50 per 100k firings Assume also the failure mode break down is: Then the Cracked/Fractured Failure rate is:.35 X 50 = 17.5/100k firings Cracked/Fractured Blades Turbine bearing Failure Pump bearing Failure Impeller Failure Turbine Seal Failure 100% 35% 25% 20% 10%

26 If the failure causes for Cracked/Fractured are determined to be: Then the Thermal Stress Failure Rate is: 0.57 X 17.5 = 10/100k firings 100% Reliability Prediction Similarity Analysis (cont’d) Fuel Turbo Pump Example

27 Failure Rate Adjustments established through: Test Results Preliminary Analyses Integrated Product Team (IPT) Input Address "high hitters" - Using Thermal Stress failure rate of 10.0/100k firing Design changes to improve reliability Cum Percent Failure Rate Improvement Reduction Lower Operating Temperatures 20% 2.00 (Test) Hollow Blades 30% (additional) 4.40 (Analysis, Expert Opinion) Material Change 20% (additional) 5.52 (Analysis) Reliability Prediction Similarity Analysis (cont’d) Fuel Turbo Pump Example

28 If no other changes are made, the FTP predicted reliability is then: = / 100k firings Reliability Prediction Similarity Analysis (cont’d) Fuel Turbo Pump Example

29 Other PRA-Related Practices Reliability Demonstration

30 Reliability Demonstration Reliability Demonstration is a reliability estimation method that primarily uses test data (objective data) to calculate demonstrated reliability with some statistical confidence. Some commonly used models and techniques for reliability demonstration include Binomial, Exponential, and Weibull models. Reliability growth techniques, such as the U.S. Army Material Systems Analysis Activity (AMSAA) and Duane models can also be used to calculate demonstrated reliability.

31 SFR Criteria is an optimization tool based on the demonstrated reliability of SSME hardware. SFR is used by the SSME Program as a quantitative probabilistic risk management tool for SSME critical hardware. SFR Criteria: Extensive fleet hot-fire experience No failures or MR history No periodic inspection Use discrete optimization for life limit determination Extend life limit up to 50% fleet leader but not to exceed the minimum run time of the six leading samples New life limit should not be less than 25% of the fleet leader Advantages include: Maximize hardware usage Use of all operational history Reliability Demonstration Example SSME Single Flight Reliability (SFR) Criteria

32 Reliability Demonstration Example SSME Single Flight Reliability (SFR) Criteria – Powerhead Assembly Example Serial Number Seconds (Partial Listing) Beta = 2.08 Powerhead Assembly LRU Code A050 25% F/L – % F/L