Presentation is loading. Please wait.

Presentation is loading. Please wait.

MITRE February 2005 Paul R. Garvey Chief Scientist Center for Acquisition and Systems Analysis Cost Risk Analysis Without Statistics!! MITREMITRE MITRE.

Similar presentations


Presentation on theme: "MITRE February 2005 Paul R. Garvey Chief Scientist Center for Acquisition and Systems Analysis Cost Risk Analysis Without Statistics!! MITREMITRE MITRE."— Presentation transcript:

1 MITRE February 2005 Paul R. Garvey Chief Scientist Center for Acquisition and Systems Analysis Cost Risk Analysis Without Statistics!! MITREMITRE MITRE Paper MP ©The MITRE Corporation, All Rights Reserved Approved For Public Release

2 MITRE 2 The Question A government agency asked the question Can a valid cost risk analysis (that is traceable and defensible) be conducted with minimal (to no) reliance on Monte Carlo simulation or other statistical methods? The question is motivated by the agencys unsatisfactory experiences in developing and implementing Monte Carlo simulations to derive risk-adjusted costs of future systems MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

3 MITRE 3 What These Charts Present These charts present an approach that addresses the question posed by the agency The approach reflects a minimum acceptable method whereby a technically valid measure of cost risk can be derived without relying on statistical methods Optional Augmentation A statistically-light augmentation to this approach is also presented that enables the agency to assess probabilities that a future systems cost will (or will not) be exceeded Best Practice Monte Carlo simulation, and the supportive statistical theory, practiced routinely by the community, remains a best technical practice for conducting a cost risk analysis MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

4 MITRE 4 The Big Picture: What is Risk? What is Uncertainty? Uncertainty vs Risk* Risk is the chance of loss or injury. In a situation that includes favorable and unfavorable events, Risk is the chance of loss or injury. In a situation that includes favorable and unfavorable events, risk is the probability an unfavorable event occurs Uncertainty is the indefiniteness about the outcome of a situation - it includes favorable and unfavorable events We analyze uncertainty for the purpose of measuring risk! In systems engineering this analysis might focus on measuring the risk of,, or ; these are examples of three unfavorable events In systems engineering this analysis might focus on measuring the risk of {failing to achieve performance objectives}, {overrunning the budgeted cost}, or {delivering the system too late to meet user needs} ; these are examples of three unfavorable events * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

5 MITRE 5 The Big Picture: What is Cost Risk Analysis and Why Conduct It? What is Cost Uncertainty Analysis?* Cost uncertainty analysis is a process of quantifying the cost impacts of associated with a systems technical definition and cost estimation methodologies What is Cost Uncertainty Analysis?* Cost uncertainty analysis is a process of quantifying the cost impacts of uncertainties associated with a systems technical definition and cost estimation methodologies What is Cost Risk Analysis?* Cost risk analysis is a process of quantifying the cost impacts of associated with a systems technical definition and cost estimation methodologies What is Cost Risk Analysis?* Cost risk analysis is a process of quantifying the cost impacts of risks associated with a systems technical definition and cost estimation methodologies What is Cost Risk? Cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded Why Conduct the Analysis? To produce a defensible assessment of the level of cost to budget such that this cost has an acceptable probability of not being exceeded MITRE Paper MP ©The MITRE Corporation, All Rights Reserved * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y.,

6 MITRE 6 A Minimum Acceptable Method …Nonstatistical Given the what and why of cost risk analysis, a minimum acceptable method is one that operates on a set of specified scenarios that, if they occurred, would result in costs higher than the level planned or budgeted These scenarios do not represent extreme worst cases; rather, they reflect a set of conditions that a decision- maker would want to have budget to guard against, should any or all of them occur For purposes of this discussion well characterize this minimum acceptable method as the Scenario-Driven approach to cost risk analysis MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

7 MITRE 7 A Minimum Acceptable Method …Nonstatistical A Scenario-Driven approach to cost risk analysis derives from what could be called sensitivity analysis, but with one difference... Instead of arbitrarily varying one or more variables to measure the sensitivity (or change) in cost, the Scenario-Driven approach specifies a well-defined set of technical and/or programmatic conditions that collectively affect a number of cost-related variables and associated WBS cost elements in a way that increase cost beyond what was planned or budgeted Multiple scenarios can be hypothesized, specified, and costed by the analysis team; the one considered most critical to guard the systems cost against is the scenario the risk analysis should be based on; call this the Prime Scenario MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

8 MITRE 8 A Minimum Acceptable Method …Nonstatistical This approach does not rely on statistics to produce a valid risk-adjusted value of a systems total cost However, with the incorporation of just two statistical inputs the Scenario-Driven approach can also provide credible percentiles of a systems total cost The following charts describe this approach and illustrate it via an example computation MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

9 MITRE 9 Scenario-Driven Cost Risk Analysis …Nonstatistical Step 1 Start with the Baseline (or Point Estimate) Cost Define Cost Sys, PE as the total cost of a system, where Cost Sys, PE is the sum of the cost element costs summed across the systems work breakdown structure (WBS) without any adjustment for risk/uncertainty; define this cost as the Baseline or Point Estimate cost Step 2 From the Prime Scenario Define Cost Sys, b* as the total cost of a system, where Cost Sys, b* is the sum of the cost element costs summed across the systems work breakdown structure (WBS) adjusted for risks you want to guard against (from a cost overrun perspective); define this cost as the cost derived from the Prime Scenario; Cost Sys, b* is the risk-adjusted total cost of the system Step 3 Derive a Measure of Cost Risk (CR) From Step 1 and Step 2 the amount of cost risk is given by CR = Cost Sys,b* - Cost Sys, PE CR = Cost Sys,b* - Cost Sys, PE CR is also a measure of the amount of reserve dollars needed to guard against the Prime Scenario MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

10 MITRE 10 Scenario-Driven Cost Risk Analysis …Nonstatistical This approach (although simple in nature) is a valid cost risk analysis; why? The process of defining these scenarios identifies the technical and cost estimation risks inherent in the baseline estimate and enables these risks to become fully visible/traceable to decision-makers Defining the Prime Scenario from the set of specified scenarios will guard for at least those sets of risks (e.g., weight growth, SLOC growth, estimation risks (e.g., levels of effort for SEPM)) considered critical to guard the baseline cost against These scenarios are essentially discrete passes through a Monte Carlo simulation; thus, this approach can be linked directly to the classical/best practice method A properly done Monte Carlo simulation must always be premised on a set of reasonably well-defined scenarios Defining these scenarios will build the rational, traceable, and analytic basis behind a derived measure of cost risk (i.e., the amount of cost risk reserve needed to guard the baseline cost against the specified risks) All the above is essentially why we do a cost risk analysis! The cost of a scenario is just the sum of the dots across the WBS cost element costs; think of this cost as a single pass through the Monte Carlo process; the above figure is from the reference below* * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

11 MITRE 11 Scenario-Driven Cost Risk Analysis …Nonstatistical What Dont You Get? Recall that, by definition, cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded Because this approach is nonstatistical, confidence measures are not produced; that is, the probabilities (i.e., the chances) that Cost Sys, PE or Cost Sys, b* will (or will not) be exceeded are not generated Can Confidence Measures be Incorporated? Yes, the following charts describe how this can be done while still staying within this general approach This is an Optional Augmentation to the Scenario-Driven approach MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

12 MITRE 12 Scenario-Driven Cost Risk Analysis OptionalAugmentation MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

13 MITRE 13 Scenario-Driven Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle Assumption 1 Assess the probability (alpha) that the true cost of the system will fall in the interval [ Cost Sys, PE, Cost Sys, b* ] [ Cost Sys, PE, Cost Sys, b* ] Assumption 2* Assume the statistical distribution for Cost Sys is uniformly distributed with probability alpha that the true cost falls in the interval [ Cost Sys, PE, Cost Sys, b* ] Let X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* Let X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* * Note: Within this assumption it is further assumed that the distribution of the total probability across the interval [a, b] is as shown to the right MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

14 MITRE 14 Scenario-Driven Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle Given the three values alpha, Cost Sys, PE, and Cost Sys, b* the minimum cost a and the maximum cost b can be derived from the formulas below Given a and b we can then determine any percentile needed from the probability formula below Prob (Cost Sys < x) = (x - a)/(b - a), where x is dollars Let X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* Let X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

15 MITRE 15 Scenario-Driven Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation I Required Inputs Dollars ($M) x This is Cost Sys ; the total cost of the system In this example, x = $34M is the derived 50th percentile cost; that is, Prob (Cost Sys < 34) = ( )/( ) = 1/2 a = $26.5Mb = $41.5M Cost Sys, b* b 1 = $40M Cost Sys, PE a 1 = $28M Alpha = 0.8 Dollars ($M) Probability that Cost Sys is less than or equal to x Outputs 0 MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

16 MITRE 16 Scenario-Driven Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II For this approach, an assessment of alpha is required In some cases, that may be hard to justify; but lets consider some reasonable possibilities From past experience, we often see the value of Cost Sys, PE fall between the 20th and 30th percentiles of the distribution function of a projects total cost Cost Sys Let X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* Let X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* So, if alpha = 0.60 then Cost Sys, PE falls at the 20th percentile; if alpha = 0.75 then Cost Sys, PE falls at the 12.5th percentile; if alpha = 0.90, then Cost Sys, PE falls at the 5th percentile MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

17 MITRE 17 Scenario-Driven Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II Thus, we can run the analysis with alpha at 0.60, 0.75, and 0.90 and develop a range of risk-adjusted costs associated with each alpha The true alpha will likely almost-always fall within this alpha range where X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* where X = Cost Sys, a 1 = Cost Sys, PE, and b 1 = Cost Sys, b* If alpha = 0.60 then Prob (Cost Sys < a 1 ) = 0.20 and Prob (Cost Sys < b 1 ) = 0.80 If alpha = 0.75 then Prob (Cost Sys < a 1 ) = and Prob (Cost Sys < b 1 ) = If alpha = 0.90 then Prob (Cost Sys < a 1 ) = 0.05 and Prob (Cost Sys < b 1 ) = 0.95 MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

18 MITRE 18 Scenario-Driven Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II 24 = 0.60 = 0.75 = 0.90 = 0.60 = 0.90 Dollars ($M) x The values in the three columns from the left-most column are derived from, and are specific to, the input parameters a 1 = 28 and b 1 = 40 given for this example. Multiply the point estimate by the values shown (in the three columns from the left- most column) to determine the costs associated with the probabilities shown in the left-most column (e.g., the 75th percentile cost, for alpha = 0.60, is (1.393)($28M) = $39M Again, suppose a 1 = $28M and b 1 = $40M; then the value of Cost Sys at (say) the 75th percentile, for alpha equal to 0.60, 0.75, and 0.90, respectively is… 75th Percentile: [$37.33M, $38M, $39M] 95th Percentile: [$40M, $41.2M, $43M] alpha = 0.90alpha = 0.60alpha = 0.75 alpha = 0.90alpha = 0.60alpha = 0.75 The decision-maker then selects a value that best reflects his/her judgement of the cost needed to guard against the cost risk posed by the Prime Scenario MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

19 MITRE 19 Scenario-Driven Cost Risk Analysis Observations This approach offers the following features Provides an analytic argument for deriving the amount of cost reserve needed to guard against a well-defined Prime Scenario Brings the discussion of scenarios and their credibility to the decision-makers; this is a more meaningful topic to focus on, instead of the statistical abstractions the classical analysis can sometimes create Does not require the use of statistical methods to develop a valid measure of cost risk reserve Percentiles (confidence measures) can be designed into the approach with a minimum set of statistical assumptions (only two required) Percentiles (as well as mean, median (50th%), variance, etc) can be calculated algebraically and thus can be executed in near-real time within a simple spreadsheet environment; Monte Carlo simulation is not needed! The approach fully supports traceability and focuses decision-makers attention on the key issues that have the potential to drive cost higher than expected The uniform distribution assumption is a conservative assumption in the sense that the distribution of probabilities is linear across the interval where its defined Does not require analysts develop probability distribution functions for all the uncertain variables in a WBS, which can be time-consuming and hard to justify Process will automatically allocate the cost risk reserve dollars into the WBS MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

20 MITRE 20 Summary The approach described herein is valid in the broad context of the principal objective of a cost risk analysis These scenarios represent a discrete set of passes through a Monte Carlo simulation Thus, youre not capturing the full range of potential cost outcomes (driven by favorable and unfavorable events) -- only the outcomes as specified by the scenarios (which are driven by unfavorable events) MITRE Paper MP ©The MITRE Corporation, All Rights Reserved

21 MITRE 21 Summary …concluded MITRE Paper MP ©The MITRE Corporation, All Rights Reserved Again, Monte Carlo simulation remains a best technical practice However, further work is needed by the community on ways to simplify the presentation of its outputs (and meaning) to senior decision-makers * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y.,


Download ppt "MITRE February 2005 Paul R. Garvey Chief Scientist Center for Acquisition and Systems Analysis Cost Risk Analysis Without Statistics!! MITREMITRE MITRE."

Similar presentations


Ads by Google