2 Road Ahead Risk Management Process Cost and Schedule Risk Estimating LikelihoodMitigationUtility and ConsequencesSensitivity Analysis
3 Reference MaterialRisk Management Guide for DOD Acquisition, 6th Ed, Ver 1.0, AugustINCOSE Systems Engineering Handbook, Ver 3, INCOSE-TP , June, 2006, Chapter 7.3
4 Why Do We Care?Every change from a current state to a future state occurs as a result of a process.The future behavior of any process is affected by uncertainties.Management of the process requires consideration of these uncertainties to minimize their influence on the desired final outcome of the process.
6 Risk vs. Issues Risk refers to a future uncertain event. If an event has occurred it is no longer uncertain. It becomes an issue or a problem.Management of either requires expenditure of resources.
7 Risk Definitions Statistics: P[Undesirable Event] Risk Management: P[Undesirable Event] Plus the Consequences of the EventINCOSE considers both Undesirable and Desirable events
8 Consequences Can Impact: Technical PerformanceKey Performance ParametersOperational capabilitySupportabilityScheduleCostThe above are not mutually independent
9 Risk CategoriesTechnical Risk – A technical requirement may not be satisfied during the life cycleCost Risk – Available budget may be exceededSchedule Risk – May fail to reach Scheduled MilestonesProgrammatic Risk – Events beyond the control of the Program Manager
10 A Risk Reporting Matrix 54High RiskModerateRiskLikelihood Level32Low Risk112345Consequence Level
12 Possible Consequence Level Criteria Technical PerformanceScheduleCost1None to Minimal2Can be toleratedLittle program impactSlip< ? moC< 1% of Budget3Moderate, Limited program impactSlip<? moSubsys slip>? Mo1%<=C<5% of Budget4Significant degradation,May jeopardize programCritical path affected5%<=C<10% of Budget5Severe degradation, Will jeopardize program successCannot meet key program milestones10% of Budget<= C
13 Risk Criteria Depend on Program Risk LevelCriteria1P<0.00012P<0.023P<0.054P<0.255P>0.25
14 Risk Reporting Illustration Risk Title (Category)CauseMitigation Approach54Likelihood Level32112345Consequence Level
15 Risk ManagementA process to minimize the adverse effects of uncertain future events on the achievement of end state objectives.Basic Tasks:Identify and characterize process properties.Decide on a course of action to minimize adverse affects of events on program objectives.Implement and control the course of action.
17 Planning Define the strategy and process to be used Establish a Risk Management Plan (RMP)TasksSchedulesReviewsReportingDefine the resources requiredPeopleFundsSpace and support resources
18 An Example RMP Format Summary 1. Introduction2. Program Summary3. Risk Management Strategy and Process4. Responsible/Executing Organization5. Risk Management Process and Procedures6. Risk Identification7. Risk Analysis8. Risk Mitigation Planning9. Risk Mitigation Implementation10. Risk Tracking
19 Events May Cause RMP Update A change in acquisition strategy,Preparation for a milestone decision,Results and findings from event–based technical reviews,An update of other program plans,Preparation for a Program Objective Memorandum submission, orA change in support strategy.
20 Risk Management Process PlanningIdentify UncertainEventsEstimateP[Occur]EstimateConsequencesFormulate AlternativeCourses of ActionNoEvaluateAlternativesChoose ApproachExecute andTrackWorking?ORYesContinueor Stop
21 Risk (Uncertain Event) Identification What can go wrong?If EVENT happens then CONSEQUENCE resultsFind everywhere Mr. Murphy can rear his ugly head!
22 What to Look AtCurrent and proposed staffing, process, design, suppliers, concept of operation, resources, interfaces, interactions, etc.Test results and failures (especially readiness results)Potential ShortfallsTrendsExternal Influences (programmatic, political)
23 Potential Root Causes of Risk Events Need (Threat)RequirementsTechnical BaselineTest & EvaluationModeling & SimulationTechnologyLogisticsProduction/FacilitiesConcurrencyIndustrial CapabilityCostManagementScheduleExternal FactorsBudgetEarned Value Realism
24 Possible Risk Management Actions AvoidRedesignChange requirementsAcceptControlExpand resourcesReduce likelihood and/or consequencesTransferBy mutual agreement to party more qualified to mitigate
25 Interactions and Consequences Schedule affects costC = A +BTBudget constrains costForces B vs T tradeoffSchedule constraint affects costCauses B increaseTechnology maturity and workforce skills affect all terms
26 Example:Program to modify an existing guided submunition Current CharacteristicsWeight 20 kgLength 50 cmDiameter 15 cmRequired CharacteristicsWeight 15 kgLength 40 cmDiameter 10 cm
31 Actions to Meet Requirements ElementDiscrepancyCauseCorrectiveActionLengthUse of discrete componentsSpecial chipsDiameterSeeker antennaNew detectorWeightDiscrete components,batteryReduce power requirements, use integrated circuits
32 Activities Required to Reduce Length IDDescriptionTime (Mo)Preceded byExpected LengthA1Analysis & Design1---50A2Breadboard Fabrication2A3TestA4Modify Breadboard0.5A5Retest43A6Admin Lead Time1.5A7Develop Chips3A5, A6, C5A8Test Chips40
33 Length Reduction Program 1.52.5A11A23A34A44.5A55A78A891210.50.531From C5
45 Effect of Variability on Schedule For the length reduction program:Average time to achieve 43 cm = 5 months95% confidence band: 4.49 –5.51 monthsAssumes Normal Distribution of activity time and 10% coefficient of variationActivity time distributions are usually triangular (a,a,c)Moves mean and right tail to the right
46 Parameter – Time Relationship 95 % Confidence RegionsTechParameterTime
47 Cost C = A + B*T C = Activity cost A = Fixed Cost B = Expenditure Rate T = Elapsed Time
50 SummaryAll activities affecting the desired end result and their interactions must be considered.Network representation takes care of thisActivities must be considered at a low enough level to permit reasonable accurate time estimates
51 Summary (cont’d) Cost for development programs is a function of time There is variability in everythingVariability can cause the critical path to changePlan for the occurrence of bad outcomes
61 Bayes’ TheoremIf Bi , i = 1, 2, …n, are mutually exclusive events, thenP[Bi] are prior, or “a priori” probabilities and must be determined prior to some experiment that results in event A based on the nature of the problem, data, experience, or subjectively based on experience.
62 Expected ValueDiscrete DistributionContinuous Distribution
63 Subjective Probability Consider the following EventsFlip a coin and let it hit the floor. Before looking at the coin, what is the probability it is “heads”?What is the probability that the coin flip before the 2009 Auburn/Alabama football game resulted in “tails”?Both events have occurred.The outcome is certain. The only uncertainty is in your mind; i.e., in your “degree of belief”
64 Subjective Probability Problems Same phrase has different connotations with different peopleInterpretation is context dependentPeople are uncomfortable doing itThere is no “correct” value
65 ExampleYou and a friend disagree on P[Your team will win its next game]. Discussion of factors, home advantage, injuries, weather, etc., does not result in agreement.How do you resolve the disagreement?
66 Assessing Subjective Probabilities Direct InquiryJust make an estimate based on knowledge and perceptionMay not be able to come up with a valueMay not have much confidence in the resultAnalysis of betsComparison of Lotteries
67 Analysis of Bets Compare two bets Look for indifference point Example LA Lakers vs Boston CelticsWhat do you think is P[Lakers Win]BetsWin X$ if LA wins, lose Y$ if LA losesLose X$ if LA wins, win Y$ if LA loses
68 Analysis of Bets (Cont’d) LA winsProcedure:Change X and Y until willing to take either bet.XBet for LALA loses-YLA wins-XBet Against LAYLA loses
69 Analysis of Bets (Cont’d) Expected ValueAt indifference EV should be the same
70 Example For LA Round 1 Against LA For LA Against LA Round 2 . Round n Indifferent
71 Reference Lottery Lottery 1 Lottery 2 (Reference) Win A if LA wins Win B if LA losesLottery 2 (Reference)Win A with probability pWin B with probability 1-p
72 Reference Lottery (Cont’d) LA winsProcedure:Change p until willing to take either bet.Outcome of lottery 2 is determined by a random process.Choose p, generate a random variate, x, from U(0,1) distribution.If x p then win A; otherwize, win BALottery 1LA losesBpALottery 2ReferenceB1-p
73 } } } Example Prefer lottery 2 Round 1 Prefer lottery 1 Home in slowly.Check for consistency with probability axioms and theorems.
74 Continuous Probability Distributions StrategiesDirect assessment of pFractile assessment of XProbability Distribution Function(Cumulative probability)
75 Direct Assessment Estimate range of values for x, xmin, xmax Pick value xi : xmin < xi < xmaxEstimateRepeat for a number of points (for 3 points pick mid range, then mid range of the two segments unless distribution is strongly skewed)Use Reference lottery approach to find indifference
80 Heuristics and Biases Representativeness Availability Anchoring and adjustingMotivational bias
81 RepresentativenessJudge that something or someone belongs to a particular categoryStereotypingInsensitive to base rates and prior probabilitiesUnreliable informationFailure to account for inherent uncertaintyMisunderstanding random processesRegression to the meanExtreme outcomes likely to be followed by one closer to the mean
82 AvailabilityJudge P[event] according to ease of recalling similar eventsInfluenced by unbalanced reportingIllusory correlation – pair of events perceived as happening together frequently
83 Anchoring and Adjusting Choose initial point then adjust about itAffects continuous distributions more that discrete probabilitiesIf initially estimate median will tend to underestimate extremes so distribution is too narrowEstimate extremes first- the “worst” extreme before the other
84 Motivational BiasIncentives often exist that motivate person to report forecasts or probabilities that do not reflect their true beliefsSalesman forecasting salesWeather forecasters forecasting rainProgram managers predicting lower cost, shorter schedule, and low risk
85 Decomposition Break problem down to make probability assessment easier Permits using people with subject matter knowledgeMore likely to get a realistic estimateLook at fault tree analysis and hazard analysisUse laws of probability to reconstruct the problemVery important concept for planning and constructing mitigation efforts
86 CoherenceAll assessed probabilities MUST obey the laws of probability
87 Using Test ResultsRequirements are often placed on characteristics that are stochasticReliabilityAccuracyCan use probability to estimate probability of not satisfying the requirement if test data existExample: Impact accuracy of a projectile specified by the standard deviation of miss distance
88 Example Requirement: Data yield sample estimate S, sample size n where RecallThenLook up alpha in a table or use the chidist(chi,dof) function in Excel to find Alpha=0.067Alpha is the risk probability
90 Value of Information Information reduces uncertainty Toss a pair of dice and do not look at resultEstimate the probability that it is 7If told it is not 5, does this change your estimate?Information has a cost and a worthShould never pay more than it is worth
91 Example*A new manufacturing process has been developed. It will cost $0.5M to implement. It will save $0.5M over investment if implemented by your current vender. There are two other vendors willing to implement the process, but both require some relief from EPA regulations in order to provide savings, which could be least double those offered by the current vendor. No relief offers savings, but not as great as keeping the current vendor. It is known the EPA is re-examining these regulations and could either provide relief, make no change, or increase the requirements. The current vendor is not impacted regardless of any changes. An increase will cause a loss by both of the other vendors which you must reimburse.* Based on an example in Making Hard Decisions, An introduction to Decision Analysis; PWS Kent, 1991; Robert T. Clemen
96 Possible Actions Obtain Information Questions: Re-estimate probabilitiesHire knowledgeable consultantKnows for sureProbably knowsQuestions:What is information worth?How reliable is the consultant’s result?
97 Perfect Information EMVPI = $1.0M - $0.58M = $0.42M No Information From previous analysis)V1$0.5Relief0.5($V2$1.0V3$1.5PerfectInformationV1$0.5Same0.3($1.0M)V2$0.2V3$0.1V1$0.5Increase0.2V2-$0.1V3-$1.0EMVPI = $1.0M - $0.58M = $0.42M(xxx)=EMV
98 Imperfect Information ?($0.58M)Relief$1.5MV3Same$0.1MIncreaseNo Consultant-$1.0MRelief$1.0M“Relief”V2Same$0.2MIncrease-$0.1MV1$0.5MRelief$1.5MV3Same$0.1MIncrease-$1.0MRelief$1.0MV2Same“Same”$0.2MIncreaseHireConsultant-$0.1MV1$0.5MRelief$1.5MV3Same$0.1MIncrease-$1.0MRelief$1.0MV2Same$0.2M“Increase”Increase-$0.1MV1$0.5M(xxx)=EMV
99 Problem Need to find conditional probabilities for chance events SolutionTotal probability lawBayes’ theorem
100 Solution Total Probability Where: R=Relief S=Same I=Increase “R” denotes predict Relief, etc.
103 Imperfect Information Relief0.8247$1.5MV3Same0.0928$0.1MIncrease0.0825No Consultant-$1.0MRelief0.8247($0.835M)$1.0M0.0928“Relief”V2Same$0.2M0.485Increase0.0825-$0.1MV1$0.5MRelief0.1667$1.5M($0.187M)V3Same0.700$0.1MIncrease0.1333-$1.0MRelief0.1667($0.293M)$1.0MV2Same“Same”0.700$0.2MHireConsultant0.300Increase0.1333-$0.1MV1$0.5M($0.822M)-($0.188)Relief0.2326$1.5MV3Same0.2093$0.1MIncrease0.5581-$1.0MEMVI=$0.822M-$0.580M=$0.242M($0.219M)Relief0.2326$1.0MV2Same0.2093$0.2M“Increase”Increase0.5581-$0.1M0.215V1$0.5M(xxx)=EMV
104 Impact on Risk Probability of $1M loss reduced from 0.2 to 0.0825 Risk changed from high yellow to low levelCan pay consultant up to $0.242M
105 ProblemUsing EMV may lead to solutions that may not be intuitively appealingExample:A1 Win $30, p=0.5 EMV = $14.50Lose $1, p=0.5A2 Win $2000, p=0.5 EMV = $500Lose $1000, p=0.5Choose A2 based on EMVWhat about Risk and Consequence?Would probably rather have A1
106 ReasonEMV is valid for the long run; i.e., multiple occurrences of the chance eventThis is a one time eventIgnores the range of possible outcomesPlay 10 timesMax loss A1 = $10Max loss A2 = $20000
107 SolutionFind a transformation of consequence into a utility measure, UMust accommodate attitude toward the risk/consequence combinationRisk averseRisk takingRisk neutral
110 Scenario Forced gamble Would you pay $x to get out of this gamble? Win $500, p=0.5Lose $500, p=0.5Would you pay $x to get out of this gamble?If so, you are risk averseExamplesInsuranceRansom
111 Comment Not everyone is risk averse Many people are risk seeking over some range and risk averse over othersDepends on wealth level (range of consequences)Risk attitudes are important in analyzing mitigation alternatives
113 TransformationUtility function transforms wealth (consequence) into a measure that accounts for risk attitudeCertainty equivalent (CE)Gamble: Win $2000, p=0.5 EMV=$990Lose $20, p=0.5Offered $300 for the gamble. Ask $301, won’t accept $299Then CE=$300 for this gambleIf CE ~ EMV thenU(CE)=EMV($)Can substitute U(x) for $xEMV – CE = Risk Premium
114 Utility Function Assessment CE Method Fix min and max values of wealthSet U(min)=0, U(max)=1Structure a lottery0.5min0.5maxCE14. Find CE1. ThenExample:
115 CE Method (Cont’d) 5. Pick another range, say CE1 – max 6. Structure a new lottery0.5CE10.5maxCE27. Set range 0 –CE18. Repeat 6. To obtain U(CE3) = 0.259. Stop or further subdivide the intervals10. Either draw the curve or fit a function
116 Probability Equivalent (PE) Method Fix max and min of wealthSet U(min) = 0, U(max) = 1Pick CE: min < CE < maxStructure lotterypmin1-pmaxCE5. Find p: CE ~ p(min) + (1-p)(max)Then:6. Repeat steps 3. – 5. For other values of CE
117 Risk Tolerance Consider the utility function R determines shape Larger R -> Flatter functionSmaller R -> more concave (risk averse)Hence, R depends on risk attitude
118 Determination of R Consider a lottery 0.5 Y A1 0.5 Y/2 A2 Find largest value of Y for which A1 > A2R = YTo find CE, Find E(U) for the decision. SolveFor x = CE
119 Caveats Utilities do not add U(A+B) = U(A)+U(B) Utilities do not express strength of preference.They only provide a numerical scale for ordering preferencesUtility functions are not the same person to personThey are subjective and express personal preferences
120 Example Revisited $ U 1.5M 1.00 1.0M 0.86 0.5M 0.65 0.2M 0.52 Relief0.5$1.0MV2Same0.3$0.2M($0.54M)Increase0.2-$0.1MRelief0.5V3$1.5M($0.58M)Same0.3$0.1MIncrease0.2-$1.0M
121 Utility Axioms Ordering and transitivity Consider events A1, A2, A3. ThenA1> A2, A2> A1, or A1~ A2If A1> A2 and A2> A3, then A1> A3Reduction of compound uncertain eventsA DM is indifferent between compound uncertain events (a complicated mix of gambles and lotteries) and a simple uncertain event as determined using standard probability manipulations.
123 Example Continued 2 E1 0.17 5 A1 E4 0.1=0.5*0.2 10.3 E5 0.25=0.5*0.5 5 0.15=0.5*0.3A2E70.066=0.33*0.210.3E80.165=0.33*0.55E90.099=0.33*0.3
124 Example Concluded 2 A1 0.166=0.10+0.066 10.3 A2 0.585=0.17+0.25+0.165 0.249=
125 Axioms Continued Continuity Substitutability A DM is indifferent between outcome A and an uncertain event with outcomes A1 and A2 where A1>A>A2. Hence we can construct a reference gamble with p(A1) and (1-p)(A2) such that the DM is indifferent between A and the gambleSubstitutabilityA DM is indifferent between an uncertain event A and one found by substituting for A an equivalent uncertain event (gamble can be substituted for CE)
126 Axioms Continued Monotonicity Invariance Boundedness Given two reference gambles having the same possible outcomes, a DM will prefer the one with higher probability of winning the preferred outcome.InvarianceOnly outcome payoffs and probabilities are needed to determine DM’s preferencesBoundednessNo outcomes are infinitely bad or infinitely bad
127 CommentsThere are some controversies and paradoxes regarding some of the axiomsIf you accept them thenThere exist U1, U2, … , Un (utilities) with associated payoffs such that the overall preference for uncertain events A and B can be determined by E(U)You should be using E(U) to make decisions (rational behavior)
128 Sensitivity AnalysisTo manage risk we need both probabilities and consequencesProbabilities, utilities, and maybe consequences are likely subjective estimatesNeed to find out how much change will impact the decision
129 Example Problem Again $ U 1.5M 1.00 1.0M 0.86 0.5M 0.65 0.2M 0.52 V1(0.5M)Relief0.50.652$1.0M0.86V2Same0.3$0.2M0.52($0.54M)Increase0.2-$0.1M0.330.638Relief0.5V3$1.5M1.00($0.58M)Same0.3$0.1M0.46Increase0.2-$1.0M0.00
132 Summary Risk management process Cost and schedule risk ModelsMonte carlo simulationEstimation of likelihoodsAnalysis of mitigation alternativesValue of informationEstimation of utility metricsEMV vs Expected utilitySensitivity analysis