1. Cost Risk Analysis Without Statistics!!
Approved For Public Release
February 2005
Paul R. Garvey
Chief Scientist
Center for Acquisition and Systems Analysis
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved
MITRE
MITRE
MITRE

2. The Question A government agency asked the question
“Can a valid cost risk analysis (that is traceable and defensible) be conducted with minimal (to no) reliance on Monte Carlo simulation or other statistical methods”?
The question is motivated by the agency’s unsatisfactory experiences in developing and implementing Monte Carlo simulations to derive “risk-adjusted” costs of future systems
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

3. What These Charts Present
These charts present an approach that addresses the question posed by the agency
The approach reflects a “minimum acceptable” method whereby a technically valid measure of cost risk can be derived without relying on statistical methods
Optional Augmentation A “statistically-light” augmentation to this approach is also presented that enables the agency to assess probabilities that a future system’s cost will (or will not) be exceeded
Best Practice
Monte Carlo simulation, and the supportive statistical theory, practiced routinely by the community, remains a best technical practice for conducting a cost risk analysis
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

4. The Big Picture: What is Risk? What is Uncertainty?
Uncertainty vs Risk*
Risk is the chance of loss or injury. In a situation that includes favorable and unfavorable events, risk is the probability an unfavorable event occurs
Uncertainty is the indefiniteness about the outcome of a situation - it includes favorable and unfavorable events
We analyze uncertainty for the purpose of measuring risk!
In systems engineering this analysis might focus on measuring the risk of {failing to achieve performance objectives}, {overrunning the budgeted cost}, or {delivering the system too late to meet user needs}; these are examples of three unfavorable events
* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y.,
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

5. The Big Picture: What is Cost Risk Analysis and Why Conduct It?
What is Cost Uncertainty Analysis?* Cost uncertainty analysis is a process of quantifying the cost impacts of uncertainties associated with a system’s technical definition and cost estimation methodologies
What is Cost Risk Analysis?* Cost risk analysis is a process of quantifying the cost impacts of risks associated with a system’s technical definition and cost estimation methodologies
What is Cost Risk? Cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded
Why Conduct the Analysis? To produce a defensible assessment of the level of cost to budget such that this cost has an acceptable probability of not being exceeded
* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y.,
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

6. A Minimum Acceptable Method…Nonstatistical
Given the “what” and “why” of cost risk analysis, a minimum acceptable method is one that operates on a set of specified scenarios that, if they occurred, would result in costs higher than the level planned or budgeted
These scenarios do not represent extreme worst cases; rather, they reflect a set of conditions that a decision-maker would want to have budget to guard against, should any or all of them occur
For purposes of this discussion we’ll characterize this minimum acceptable method as the “Scenario-Driven” approach to cost risk analysis
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

7. A Minimum Acceptable Method…Nonstatistical
A Scenario-Driven approach to cost risk analysis derives from what could be called “sensitivity analysis”, but with one difference...
Instead of arbitrarily varying one or more variables to measure the sensitivity (or change) in cost, the Scenario-Driven approach specifies a well-defined set of technical and/or programmatic conditions that collectively affect a number of cost-related variables and associated WBS cost elements in a way that increase cost beyond what was planned or budgeted
Multiple scenarios can be hypothesized, specified, and costed by the analysis team; the one considered most critical to guard the system’s cost against is the scenario the risk analysis should be based on; call this the Prime Scenario
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

8. A Minimum Acceptable Method…Nonstatistical
This approach does not rely on statistics to produce a valid risk-adjusted value of a system’s total cost
However, with the incorporation of just two statistical inputs the Scenario-Driven approach can also provide credible percentiles of a system’s total cost
The following charts describe this approach and illustrate it via an example computation
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

9. “Scenario-Driven” Cost Risk Analysis…Nonstatistical
Step 1 Start with the Baseline (or Point Estimate) Cost
Define CostSys, PE as the total cost of a system, where CostSys, PE is the sum of the cost element costs summed across the system’s work breakdown structure (WBS) without any adjustment for risk/uncertainty; define this cost as the Baseline or Point Estimate cost
Step 2 From the Prime Scenario
Define CostSys, b* as the total cost of a system, where CostSys, b* is the sum of the cost element costs summed across the system’s work breakdown structure (WBS) adjusted for risks you want to guard against (from a cost overrun perspective); define this cost as the cost derived from the Prime Scenario; CostSys, b* is the “risk-adjusted” total cost of the system
Step 3 Derive a Measure of Cost Risk (CR)
From Step 1 and Step 2 the amount of cost risk is given by
CR = CostSys,b* - CostSys, PE
CR is also a measure of the amount of reserve dollars needed to guard against the Prime Scenario
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

10. “Scenario-Driven” Cost Risk Analysis…Nonstatistical
This approach (although simple in nature) is a valid cost risk analysis; why?
The process of defining these scenarios identifies the technical and cost estimation risks inherent in the baseline estimate and enables these risks to become fully visible/traceable to decision-makers
Defining the “Prime Scenario” from the set of specified scenarios will guard for at least those sets of risks (e.g., weight growth, SLOC growth, estimation risks (e.g., levels of effort for SEPM)) considered critical to guard the baseline cost against
These scenarios are essentially discrete passes through a Monte Carlo simulation; thus, this approach can be linked directly to the classical/best practice method
A properly done Monte Carlo simulation must always be premised on a set of reasonably well-defined scenarios
Defining these scenarios will build the rational, traceable, and analytic basis behind a “derived” measure of cost risk (i.e., the amount of cost risk reserve needed to guard the baseline cost against the specified risks)
All the above is essentially why we do a cost risk analysis!
The cost of a scenario is just the sum of the “dots” across the WBS cost element costs; think of this cost as a single pass through the Monte Carlo process; the above figure is from the reference below*
* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y.,
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

11. “Scenario-Driven” Cost Risk Analysis…Nonstatistical
What Don’t You Get?
Recall that, by definition, cost risk is a measure of the chance that, due to unfavorable events, the planned or budgeted cost of a project will be exceeded
Because this approach is nonstatistical, confidence measures are not produced; that is, the probabilities (i.e., the chances) that CostSys, PE or CostSys, b* will (or will not) be exceeded are not generated
Can Confidence Measures be Incorporated?
Yes, the following charts describe how this can be done while still staying within this general approach
This is an Optional Augmentation to the Scenario-Driven approach
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

12. “Scenario-Driven” Cost Risk Analysis
Optional
Augmentation
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

13. “Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle
Assumption 1 Assess the probability (alpha) that the “true” cost of the system will fall in the interval
[ CostSys, PE , CostSys, b* ]
Assumption 2* Assume the statistical distribution for CostSys is uniformly distributed with probability alpha that the “true” cost falls in the interval
* Note: Within this assumption it is further assumed that the distribution of the total probability across the interval [a, b] is as shown to the right
Let X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b*
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

14. Prob (CostSys < x) = (x - a)/(b - a), where x is dollars
“Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle
Given the three values alpha, CostSys, PE , and CostSys, b* the minimum cost a and the maximum cost b can be derived from the formulas below
Given a and b we can then determine any percentile needed from the probability formula below
Prob (CostSys < x) = (x - a)/(b - a), where x is dollars
Let X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b*
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

15. “Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation I
This is CostSys; the total
cost of the system
Alpha = 0.8
Dollars ($M)
Outputs
a = $26.5M
CostSys, PE
CostSys, b*
b = $41.5M
a1 = $28M
b1 = $40M
Probability that CostSys is less than or equal to x
Required Inputs x
Dollars ($M)
In this example, x = $34M is the derived 50th percentile cost; that is,
Prob (CostSys < 34) = ( )/( ) = 1/2
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

16. “Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II
For this approach, an assessment of “alpha” is required
In some cases, that may be hard to justify; but let’s consider some reasonable possibilities
From past experience, we often see the value of CostSys, PE fall between the 20th and 30th percentiles of the distribution function of a project’s total cost CostSys
Let X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b*
So, if alpha = 0.60 then CostSys, PE falls at the 20th percentile; if alpha = 0.75 then CostSys, PE falls at the 12.5th percentile; if alpha = 0.90, then CostSys, PE falls at the 5th percentile
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

17. “Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II
If alpha = 0.60 then Prob (CostSys < a1) = and Prob (CostSys < b1) = 0.80
If alpha = 0.75 then Prob (CostSys < a1) = and Prob (CostSys < b1) = 0.875
If alpha = 0.90 then Prob (CostSys < a1) = and Prob (CostSys < b1) = 0.95
where X = CostSys , a1 = CostSys, PE , and b1 = CostSys, b*
Thus, we can run the analysis with alpha at 0.60, 0.75, and 0.90 and develop a range of “risk-adjusted” costs associated with each alpha
The “true” alpha will likely “almost-always” fall within this alpha range
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

18. “Scenario-Driven” Cost Risk Analysis Optional Augmentation: Incorporating a Statistical Angle: Example Calculation II
Again, suppose a1 = $28M and b1 = $40M; then the value of CostSys at (say) the 75th percentile, for alpha equal to 0.60, 0.75, and 0.90, respectively is…
75th Percentile: [$37.33M, $38M, $39M]
95th Percentile: [$40M, $41.2M, $43M]
alpha = 0.90
alpha = 0.75
alpha = 0.60
The decision-maker then selects a value that best reflects his/her judgement of the cost needed to guard against the cost risk posed by the Prime Scenario
alpha = 0.90
alpha = 0.75
alpha = 0.60
The values in the three columns from the left-most column are derived from, and are specific to, the input parameters a1 = 28 and b1 = 40 given for this example.
Multiply the point estimate by the values shown (in the three columns from the left-most column) to determine the costs associated with the probabilities shown in the left-most column (e.g., the 75th percentile cost, for alpha = 0.60, is (1.393)($28M) = $39M
a = 0.60
a = 0.75
a = 0.90
a = 0.90
a = 0.75
a = 0.60
37.33 38 39
a = 0.60
a = 0.90
Dollars ($M) x 24
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

19. “Scenario-Driven” Cost Risk Analysis Observations
This approach offers the following features
Provides an analytic argument for deriving the amount of cost reserve needed to guard against a well-defined “Prime Scenario”
Brings the discussion of “scenarios” and their credibility to the decision-makers; this is a more meaningful topic to focus on, instead of the statistical abstractions the classical analysis can sometimes create
Does not require the use of statistical methods to develop a valid measure of cost risk reserve
Percentiles (confidence measures) can be designed into the approach with a minimum set of statistical assumptions (only two required)
Percentiles (as well as mean, median (50th%), variance, etc) can be calculated algebraically and thus can be executed in near-real time within a simple spreadsheet environment; Monte Carlo simulation is not needed!
The approach fully supports traceability and focuses decision-makers attention on the key issues that have the potential to drive cost higher than expected
The uniform distribution assumption is a conservative assumption in the sense that the distribution of probabilities is linear across the interval where it’s defined
Does not require analysts develop probability distribution functions for all the uncertain variables in a WBS, which can be time-consuming and hard to justify
Process will automatically allocate the cost risk reserve dollars into the WBS
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

20. Summary
The approach described herein is valid in the broad context of the principal objective of a cost risk analysis
These scenarios represent a discrete set of “passes” through a Monte Carlo simulation
Thus, you’re not capturing the full range of potential cost outcomes (driven by favorable and unfavorable events) --only the outcomes as specified by the scenarios (which are driven by unfavorable events)
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved

21. Summary…concluded
Again, Monte Carlo simulation remains a best technical practice
However, further work is needed by the community on ways to simplify the presentation of its outputs (and meaning) to senior decision-makers
* Garvey, Paul R., Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, 2000, Marcel Dekker, Inc., New York, N.Y.,
MITRE Paper MP
©The MITRE Corporation, All Rights Reserved