1. © The Aerospace Corporation 2012 Logic -Quantitative Framework for Decisionmaker's Management of Mission Failure Risk USC – CSSE Annual Research Review Workshop 7 March 2012 Dr. Sergio Guarro Distinguished Engineer, The Aerospace Corporation

2. Background and Context of The Aerospace Corporation Mission Assurance and Risk Framework Space missions are unforgiving –The rule of the game is essentially “one strike and you are out” – i.e., minimal possibility exists for remedying problems during mission execution –Volume and mass constraints also limit the amount of redundancy that can be used as insurance against failures The possibility of failure must be understood and managed at the lowest levels of system design detail Because of the above Mission Assurance and Mission Risk Assessment processes are given great attention and priority in the range of activities our company executes on behalf of our U.S. Government space program Customers –Comprehensive Program Offices’ mission assurance task plans and assessment processes –Special issues addressed with specialized Engineering & Technology Group support –Aerospace specialists’ analyses in support of Customers’ independent review team assessments 2

3. APR / ASMR Framework & Process The Aerospace Corporation (“Aerospace”) President Review / Senior Management Review (APR / ASMR) process is the concluding synthesis of a full cycle of assurance and risk assessment applied to supported National Security Space (NSS) programs, to provide the decision-maker with the analytical means to judge and manage the risk of mission failure Structured integration of Risk Assessment (RA) and Management (RM) information produced by program contractor(s) and Government / Aerospace Program Office is key to success of APR/ASMR process The logic – quantitative risk framework presented here is the result of the most recent development to provide reference guidance for the APR / ASMR risk assessment processes –The guidance is documented in a corporate Technical Instruction published by The Aerospace Corporation Corporate Chief Engineer Office and supported by more detailed documentation produced by The Aerospace Corporation Systems Engineering Division 3 PO RM Process Indpdt. Review Team RA Process APR / ASMR RA Process Contractor RM Process Indpdt. Program Reviews Focus on Mission Risk Only Focus on Mission Risk Only Address both Programmatic & Mission Risk

4. Objectives and Flow of Logic-Quantitative Risk Framework Key objectives A.Clear identification of key factors and events that can determine a mission impact B.Assessment of risk in objective probability and mission-impact dimensions Avoid qualitative definition of likelihood and consequences that are intrinsically subject to different interpretations by different audiences C.Separation of risk definition and assessment from decision process Value judgment of risk is the decision-maker’s, not the assessor’s, responsibility Typical execution flow 4 Program Office Eng. &Tech. Group MA Plan & Scope Preliminary Identification & Evaluation Risk Screening Risk Scenario Definition Risk Assessment & Rating Risk Communication & Decision Support

5. Risk Identification Using Mission Assurance Baseline Risk identification proceeds from the basic concept of risk as deviation from “mission assurance baseline” (MAB): –Each space system mission item (SSMI) within the assessment scope is evaluated from this perspective –A potential SSMI risk item is identified as a significant deviation from the desired level of quality in a set of reference mission assurance attributes –Guidance document s define the set of attributes to be evaluated and the severity criteria to determine whether any existing deviations are significant enough to call for the formal definition of an associated risk This risk identification concept is the application of a general concept that relates risk directly to the Aerospace MA (Mission Assurance) processes 5

6. Risk Item Screening Apply filter to preliminarily-identified potential risks Apply full assessment and quantification technique to mission impacting major technical risks 6 Preliminary Risk Identification 100s of non- quantitative potential risks preliminarily identified (e.g., in MA Baseline task executions) no yes Cost & Schedule Risks Mission Impact? no yes Lower Level Issues yes Lower Severity Risks Technical risks 10-20 Mission Impacting Major Technical Risks no Define & Assess Risk Scenario Significant Deviation from Baseline? Technical Impact?

7. A risk scenario is defined to initiate the analysis / assessment portion of the process for post-screen mission risks Definition: A RISK SCENARIO is a system or mission condition that can be formally described as a cause- effect sequence of events the occurrence of which may cause a mission risk impact and associated consequences to be realized. The reference risk scenario identifies in logic event sequence diagram (ESD) format the key chance events / conditions that may affect the outcome of a given risk in terms of probability and consequence severity –This may include risk control measures expressly introduced by a program to counter an identified risk: preventive control measures (PCMs), when executed successfully, eliminate altogether the potential mission impact of a given risk mitigative control measures (MCMs), when executed successfully, reduce the potential mission impact of a given risk by some predicted amount that can be quantified as a consequence reduction factor Risk Scenario Definition 7 SCENARIO EVENT SEQUENCE DIAGRAM (ESD) - including PCM & MCM events Initiating Event occurs Intermediate Event Y occurs PCMs are present and successful Intermediate Event X occurs MCMs are present and successful Unmitigated Mission Impact is realized Mitigated Mission Impact is realized No Mission Impact is realized no yes no yes

8. Risk Rating Once a reference risk scenario has been defined and expressed in standard ESD form risk can be assessed by estimating: Likelihood / probability of initiating event Conditional likelihood / probability of intermediate events –Including probability of success of PCMs and MCMs Severity / magnitude of mission performance shortfall resulting from any mission impact scenario outcomes Performance shortfall reduction factors associated with MCM-event successful outcomes The guidance documentation provides the simple formulations by which risk scenarios can be quantified and rated in summary “probability of consequence severity” form, using the above ESD quantification parameters 8

9. Legend Performance Parameter Shortfall (% of required value) Performance Requirement Iso-consequence calibration line 1 No Missn. Value 0.250.500.75 Mission Shortfall Metric (MSM) Image Resolution Shortfall 40%10 %20%30% 0 % 0 Full Missn. Value Data Rate Shortfall 80%20 %40%60% 0 % Rating of Multiple Performance Consequence Effects When a risk involves consequences in multiple mission performance dimensions a combined Mission Shortfall Metric (MSM) needs to be developed –This can be done by mapping hypothetical shortfall magnitudes relative to individual key performance parameters into a single MSM scale, i.e., essentially defining a simple “mission utility function” (in the potential shortfall direction) 9

10. Risk Communication and Decision Support The recommended format of risk communication is a “probability vs. consequence severity” risk map on which appropriate areas of risk have been pre-identified for reference according to decision makers’ input and directives Uncertainty in both probability and consequence magnitude is also displayed 10 Estimates of individual Mission Risks with low uncertainty Estimates of individual Mission Risks with significant uncertainty

11. Use in Decision-Making: Power Distribution Shorts Scenario Example Risks flow from initiator through intermediate events to impacts The initiator is not the risk Include and show “delta effect” of any preventive or mitigative control measures (PCMs, MCMs) Benefits: Easier to understand and more thorough “risk statement” Clearly identifies key events and factors, which strongly influence risk outcome Shows effectiveness of prevention and mitigation Makes it easier to resolve disputes 11 Short Occurs Short Occurs Short Is in Unprotected Section Short Load > 20 Amps Mission Loss PCM1 Add Extra Insulation PCM1 Add Extra Insulation PCM2 Add Diode Protection PCM2 Add Diode Protection MCM1 Sectorize Solar Panel MCM1 Sectorize Solar Panel Wiring Insulation Cracked / Frayed Solar Panel Wiring Insulation Cracked / Frayed Short Load 6 to 20 Amps Short Load < 6 Amps Mission Degradation No / Minor Impact Potential Controls

12. Power Shorts Scenario Example Details 12 Scenario Outcome Probability of Scenario Outcome Mission ShortfallProbability Deviation from SPF Control Requirements No Impact 1x1% No Impact 197.22% Short Develops NO (P1) YES No Impact 2x2% 11% to 64% Mission Shortfall 0.11% Short Is in Protected Section NO (P2) YES A Mission Loss2.68% Short to Structure vs. Wire STRUCTURE (P3) WIRE B A No Impact 3x3% Open Circuit Follows YES (P4) NO Large Bus Hot Load a to b A z1% Addtl. Short from Melting Insulation in Yoke YES (P5) NO Bus Hot Load c to d A y1% B No Impact 4x4% Open Circuit Follows YES (P6) NO Large Hot Load e to f A z2% Additl. Short from Melting Insulation in Nearby Wire YES (P7) NO Hot Load ~ g A z3% Total No Impact 97.22% Mission ShortfallProbability No Impact 1XX% 11% to 64% Mission Shortfall YY% Mission LossZZ%

13. Power Shorts Scenario Results RISK SCENARIO OUTCOMESMISSION OUTCOMES Large Amp Short to Bus Moderate Amp Short to Bus Moderate Amp Short to Wire(s) Large Amp Short to Wire(s) No Significant Mission Impact Mission Performance Shortfall Total Mission Loss [a to b A][c to d A][e to f A][~ g A] [m to n% shortfall] Probability Distribution Parameters Mean------- 5 th Percentile------- Median (50th Percentile) ------- 95th Percentile------- Assessment results suggested that some risk control measures would be warranted, if their introduction were technically feasible

14. 14 Defined and formulated to support Decision-makers’ assessment and management of risk of mission failure –Clear, unequivocal definition / description of all “selected risks” –“Reference Scenario” Format –Distinction between assessment, display/communication, and decision- support aspects of risk process –Assessment via objective, quantifiable metrics –Quantification recommended for objectivity, not to project impression of precision Strong recommendation to explicitly display assessment uncertainty In Summary: Key Points of Logic-Quantitative Risk Framework

15. 15 Backup Charts

16. Example of MA Baseline Attributes for Risk Identification 16

17. Initiating Event Identification in Risk Scenario ESD The initiating event in a risk scenario is identified according to the nature of the baseline deviation(s) initially identifying the risk 17 SSMI BASELINE FACTORS TO BE EXAMINED TO DEFINE RISK SCENARIO INITIAL CONDITION Requirements Deviations ? Design Deviations ? Manufacturing & Assembly Deviations ? IT & E Deviations ? Operational Readiness Deviations ? MA Discipline Specs & Stds Deviations? If evidence of deviations exist, is it in process or product attributes ? If any deviations exist, is their magnitude moderate, significant, or large (M, S, or L) ? RISK-SCENARIO INITIAL CONDITION DEFINED IN TERMS OF ANSWERS TO ABOVE QUESTIONS

18. Examples of ESD Templates Provided in Risk Guidance Document SSMI product exhibits [ M / S / L ] deviation from [req./des./… ] baseline Is SSMI deviation fully controlled by built-in system design features (e.g., redundancy, operational options, etc.) ? Is SSMI deviation fully controlled by PCMs added after risk identification ? Is SSMI deviation mitigated by MCMs added after risk identification ? Unmitigated Mission Shortfalls are realized Less severe Mission Shortfalls are realized No Mission Shortfalls are realized no yes no yes ESD Template for Risk Scenario Driven by SSMI Product Attribute Deviation 18 SSMI process exhibits deviation from baseline Is a SSMI product deviation from baseline produced as a result ? Is SSMI product deviation “moderate” (M)? Enter “product deviation” ESD w/ “S” deviation condition Enter product deviation ESD w/ “M” deviation condition No Mission Shortfalls are realized yes no yes no yes Is SSMI product deviation “significant” (S)? Enter “product deviation” ESD w/ “l” deviation condition no ESD Template for Risk Scenario Driven by SSMI Process Attribute Deviation