Cost Risk Analysis Without Statistics!! Approved For Public Release February 2005 Paul R. Garvey Chief Scientist Center for Acquisition and Systems Analysis MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved MITRE MITRE MITRE

The Question A government agency asked the question “Can a valid cost risk analysis (that is traceable and defensible) be conducted with minimal (to no) reliance on Monte Carlo simulation or other statistical methods”? The question is motivated by the agency’s unsatisfactory experiences in developing and implementing Monte Carlo simulations to derive “risk-adjusted” costs of future systems MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

What These Charts Present These charts present an approach that addresses the question posed by the agency The approach reflects a “minimum acceptable” method whereby a technically valid measure of cost risk can be derived without relying on statistical methods Optional Augmentation A “statistically-light” augmentation to this approach is also presented that enables the agency to assess probabilities that a future system’s cost will (or will not) be exceeded Best Practice Monte Carlo simulation, and the supportive statistical theory, practiced routinely by the community, remains a best technical practice for conducting a cost risk analysis MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

The Big Picture: What is Risk? What is Uncertainty? Uncertainty vs Risk* Risk is the chance of loss or injury. In a situation that includes favorable and unfavorable events, risk is the probability an unfavorable event occurs Uncertainty is the indefiniteness about the outcome of a situation - it includes favorable and unfavorable events We analyze uncertainty for the purpose of measuring risk! In systems engineering this analysis might focus on measuring the risk of {failing to achieve performance objectives}, {overrunning the budgeted cost}, or {delivering the system too late to meet user needs}; these are examples of three unfavorable events * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

The Big Picture: What is Cost Risk Analysis and Why Conduct It? What is Cost Uncertainty Analysis?* Cost uncertainty analysis is a process of quantifying the cost impacts of uncertainties associated with a system’s technical definition and cost estimation methodologies What is Cost Risk Analysis?* Cost risk analysis is a process of quantifying the cost impacts of risks associated with a system’s technical definition and cost estimation methodologies What is Cost Risk? Cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded Why Conduct the Analysis? To produce a defensible assessment of the level of cost to budget such that this cost has an acceptable probability of not being exceeded * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

A Minimum Acceptable Method…Nonstatistical Given the “what” and “why” of cost risk analysis, a minimum acceptable method is one that operates on a set of specified scenarios that, if they occurred, would result in costs higher than the level planned or budgeted These scenarios do not represent extreme worst cases; rather, they reflect a set of conditions that a decision-maker would want to have budget to guard against, should any or all of them occur For purposes of this discussion we’ll characterize this minimum acceptable method as the “Scenario-Driven” approach to cost risk analysis MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

A Minimum Acceptable Method…Nonstatistical A Scenario-Driven approach to cost risk analysis derives from what could be called “sensitivity analysis”, but with one difference... Instead of arbitrarily varying one or more variables to measure the sensitivity (or change) in cost, the Scenario-Driven approach specifies a well-defined set of technical and/or programmatic conditions that collectively affect a number of cost-related variables and associated WBS cost elements in a way that increase cost beyond what was planned or budgeted Multiple scenarios can be hypothesized, specified, and costed by the analysis team; the one considered most critical to guard the system’s cost against is the scenario the risk analysis should be based on; call this the Prime Scenario MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

A Minimum Acceptable Method…Nonstatistical This approach does not rely on statistics to produce a valid risk-adjusted value of a system’s total cost However, with the incorporation of just two statistical inputs the Scenario-Driven approach can also provide credible percentiles of a system’s total cost The following charts describe this approach and illustrate it via an example computation MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis…Nonstatistical Step 1 Start with the Baseline (or Point Estimate) Cost Define CostSys, PE as the total cost of a system, where CostSys, PE is the sum of the cost element costs summed across the system’s work breakdown structure (WBS) without any adjustment for risk/uncertainty; define this cost as the Baseline or Point Estimate cost Step 2 From the Prime Scenario Define CostSys, b* as the total cost of a system, where CostSys, b* is the sum of the cost element costs summed across the system’s work breakdown structure (WBS) adjusted for risks you want to guard against (from a cost overrun perspective); define this cost as the cost derived from the Prime Scenario; CostSys, b* is the “risk-adjusted” total cost of the system Step 3 Derive a Measure of Cost Risk (CR) From Step 1 and Step 2 the amount of cost risk is given by CR = CostSys,b* - CostSys, PE CR is also a measure of the amount of reserve dollars needed to guard against the Prime Scenario MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis…Nonstatistical This approach (although simple in nature) is a valid cost risk analysis; why? The process of defining these scenarios identifies the technical and cost estimation risks inherent in the baseline estimate and enables these risks to become fully visible/traceable to decision-makers Defining the “Prime Scenario” from the set of specified scenarios will guard for at least those sets of risks (e.g., weight growth, SLOC growth, estimation risks (e.g., levels of effort for SEPM)) considered critical to guard the baseline cost against These scenarios are essentially discrete passes through a Monte Carlo simulation; thus, this approach can be linked directly to the classical/best practice method A properly done Monte Carlo simulation must always be premised on a set of reasonably well-defined scenarios Defining these scenarios will build the rational, traceable, and analytic basis behind a “derived” measure of cost risk (i.e., the amount of cost risk reserve needed to guard the baseline cost against the specified risks) All the above is essentially why we do a cost risk analysis! The cost of a scenario is just the sum of the “dots” across the WBS cost element costs; think of this cost as a single pass through the Monte Carlo process; the above figure is from the reference below* * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis…Nonstatistical What Don’t You Get? Recall that, by definition, cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded Because this approach is nonstatistical, confidence measures are not produced; that is, the probabilities (i.e., the chances) that CostSys, PE or CostSys, b* will (or will not) be exceeded are not generated Can Confidence Measures be Incorporated? Yes, the following charts describe how this can be done while still staying within this general approach This is an Optional Augmentation to the Scenario-Driven approach MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Optional Augmentation MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle Assumption 1 Assess the probability (alpha) that the “true” cost of the system will fall in the interval [ CostSys, PE , CostSys, b* ] Assumption 2* Assume the statistical distribution for CostSys is uniformly distributed with probability alpha that the “true” cost falls in the interval * Note: Within this assumption it is further assumed that the distribution of the total probability across the interval [a, b] is as shown to the right Let X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b* MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

Prob (CostSys < x) = (x - a)/(b - a), where x is dollars “Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle Given the three values alpha, CostSys, PE , and CostSys, b* the minimum cost a and the maximum cost b can be derived from the formulas below Given a and b we can then determine any percentile needed from the probability formula below Prob (CostSys < x) = (x - a)/(b - a), where x is dollars Let X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b* MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation I This is CostSys; the total cost of the system Alpha = 0.8 Dollars ($M) Outputs a = $26.5M CostSys, PE CostSys, b* b = $41.5M a1 = $28M b1 = $40M Probability that CostSys is less than or equal to x Required Inputs x Dollars ($M) In this example, x = $34M is the derived 50th percentile cost; that is, Prob (CostSys < 34) = (34 - 26.5)/(41.5 - 26.5) = 1/2 MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II For this approach, an assessment of “alpha” is required In some cases, that may be hard to justify; but let’s consider some reasonable possibilities From past experience, we often see the value of CostSys, PE fall between the 20th and 30th percentiles of the distribution function of a project’s total cost CostSys Let X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b* So, if alpha = 0.60 then CostSys, PE falls at the 20th percentile; if alpha = 0.75 then CostSys, PE falls at the 12.5th percentile; if alpha = 0.90, then CostSys, PE falls at the 5th percentile MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II If alpha = 0.60 then Prob (CostSys < a1) = 0.20 and Prob (CostSys < b1) = 0.80 If alpha = 0.75 then Prob (CostSys < a1) = 0.125 and Prob (CostSys < b1) = 0.875 If alpha = 0.90 then Prob (CostSys < a1) = 0.05 and Prob (CostSys < b1) = 0.95 where X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b* Thus, we can run the analysis with alpha at 0.60, 0.75, and 0.90 and develop a range of “risk-adjusted” costs associated with each alpha The “true” alpha will likely “almost-always” fall within this alpha range MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II Again, suppose a1 = $28M and b1 = $40M; then the value of CostSys at (say) the 75th percentile, for alpha equal to 0.60, 0.75, and 0.90, respectively is… 75th Percentile: [$37.33M, $38M, $39M] 95th Percentile: [$40M, $41.2M, $43M] alpha = 0.90 alpha = 0.75 alpha = 0.60 The decision-maker then selects a value that best reflects his/her judgement of the cost needed to guard against the cost risk posed by the Prime Scenario alpha = 0.90 alpha = 0.75 alpha = 0.60 The values in the three columns from the left-most column are derived from, and are specific to, the input parameters a1 = 28 and b1 = 40 given for this example. Multiply the point estimate by the values shown (in the three columns from the left-most column) to determine the costs associated with the probabilities shown in the left-most column (e.g., the 75th percentile cost, for alpha = 0.60, is (1.393)($28M) = $39M a = 0.60 a = 0.75 a = 0.90 a = 0.90 a = 0.75 a = 0.60 37.33 38 39 a = 0.60 a = 0.90 Dollars ($M) x 24 MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

“Scenario-Driven” Cost Risk Analysis Observations This approach offers the following features Provides an analytic argument for deriving the amount of cost reserve needed to guard against a well-defined “Prime Scenario” Brings the discussion of “scenarios” and their credibility to the decision-makers; this is a more meaningful topic to focus on, instead of the statistical abstractions the classical analysis can sometimes create Does not require the use of statistical methods to develop a valid measure of cost risk reserve Percentiles (confidence measures) can be designed into the approach with a minimum set of statistical assumptions (only two required) Percentiles (as well as mean, median (50th%), variance, etc) can be calculated algebraically and thus can be executed in near-real time within a simple spreadsheet environment; Monte Carlo simulation is not needed! The approach fully supports traceability and focuses decision-makers attention on the key issues that have the potential to drive cost higher than expected The uniform distribution assumption is a conservative assumption in the sense that the distribution of probabilities is linear across the interval where it’s defined Does not require analysts develop probability distribution functions for all the uncertain variables in a WBS, which can be time-consuming and hard to justify Process will automatically allocate the cost risk reserve dollars into the WBS MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

Summary The approach described herein is valid in the broad context of the principal objective of a cost risk analysis These scenarios represent a discrete set of “passes” through a Monte Carlo simulation Thus, you’re not capturing the full range of potential cost outcomes (driven by favorable and unfavorable events) --only the outcomes as specified by the scenarios (which are driven by unfavorable events) MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved

Summary…concluded Again, Monte Carlo simulation remains a best technical practice However, further work is needed by the community on ways to simplify the presentation of its outputs (and meaning) to senior decision-makers * Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y., 10016. http://www.sceaonline.net/content.asp?contentid=203 MITRE Paper MP 050000001 ©The MITRE Corporation, All Rights Reserved