Presentation is loading. Please wait.

Presentation is loading. Please wait.

Analysis and Implementation of Cost Estimating Risk in the Ballistic Missile Defense Organization (BMDO) Risk Model A Study of Distribution Presented to.

Published byWinfred Lawson Modified over 8 years ago

Similar presentations

Presentation on theme: "Analysis and Implementation of Cost Estimating Risk in the Ballistic Missile Defense Organization (BMDO) Risk Model A Study of Distribution Presented to."— Presentation transcript:

1 Analysis and Implementation of Cost Estimating Risk in the Ballistic Missile Defense Organization (BMDO) Risk Model A Study of Distribution Presented to the 1999 Joint ISPA/SCEA International Conference 8-11 June 1999 Prepared for: Deputy for Program Operations THE BALLISTIC MISSILE DEFENSE ORGANIZATION 7100 Defense Pentagon Washington, D.C. 20301-7100 Prepared by: Heather F. Chelson, TASC, Inc. Jessica R. Ayers, TASC, Inc. Richard L. Coleman, TASC, Inc. Nadiyeh Shahidi, TASC, Inc. Donna M. Snead, BMDO hfchelson@tasc.com 703-633-8300 x4821 jayers@bmdo.mcri.com 703-416-9500

2 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95002 1999 Joint ISPA/SCEA Int’l Conf. Outline # Risk Definitions # Purpose # Approach # New SAR Database # Analysis # First, Second, and Third Moments # Distribution # Correlation # Comparison to previous study # Moments and Distribution # Interpretation of Bias # Conclusion

3 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95003 1999 Joint ISPA/SCEA Int’l Conf. Risk Definitions # Cost Growth = Cost Estimating Growth + Sked/Tech Growth + Requirements Growth + Threat Growth # Cost Risk = Cost Estimating Risk + Sked/Technical Risk + Requirements Risk + Threat Risk # Cost Estimating Risk: Risk due to cost estimating errors, and the statistical uncertainty in the estimate # Schedule/Technical Risk: Risk due to inability to conquer problems posed by the intended design in the current CARD # Requirements Risk: Risk due to as-yet-unseen design shift from the current CARD arising due to CARD shortfalls # Risk arising due to the inability of the intended design to perform the (unchanged) intended mission # “Not understanding the solution” # Threat Risk: Due to as-yet-unrevealed threat shift from the current STAR # “Not understanding the problem”

4 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95004 1999 Joint ISPA/SCEA Int’l Conf. Example Cost Estimate with Risk 0 50 100 150 Initial Point Estimate Add Cost Estimating Risk Add Sched/Tech Risk S/T Risk CE Risk Init Pt Est $ 3% 20% 23%

5 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95005 1999 Joint ISPA/SCEA Int’l Conf. Purpose # To review the Cost Estimating (CE) risk component of BMDO’s Cost Risk Methodology # The BMDO risk model currently performs risk estimation for 12 major BMDO elements including, e.g., PAC-3, THAAD, Navy Theater- Wide TBMD, and elements of the National Missile Defense Program # To gain insight into CE Risk # Large dollars (1/7 th of total risk), but small insight # CE risk is one element of BMDO’s total cost risk. The current model uses # CE risk ~ 3.4% # Schedule/technical risk ~ 20% # This results in a total risk of ~ 23%. 1 Each of these factors is passed through the model with functional correlation. 2 1 Distribution and Correlation of Sources of Cost Risk, G.E. Hartigan, J.R. Ayers, and R.L. Coleman, May 15 1997. 2 An Overview of Correlation and Functional Dependencies in Cost Risk and Uncertainty Analysis, R. L. Coleman, S. S. Gupta, DoDCAS 1994, and Cost Risk Estimates Incorporating Functional Correlation, Acquisition Phase Relationships, and Realized Risk, by R.L. Coleman, S.S. Gupta, J.R. Ayers, and G.E. Hartigan, May 15 1997

6 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95006 1999 Joint ISPA/SCEA Int’l Conf. Approach # Update to the previous analysis 1 of BMDO’s Cost Estimating risk. To review: # Found that CE and S/T risk were uncorrelated # Internal variance, a proxy for CE risk was normally distributed # Update the model with new findings # Current model reflects the above research: Cost Estimating Risk 3.4% mean, 14.6% St Dev CE Risk is Normally distributed CE Risk is constant by phase CE Risk is computed once, and passed forward using Functional Correlation # Create new SAR database: increase size and stratification 1 Distribution and Correlation of Sources of Cost Risk, 15 May 1997, by Hartigan, Ayers and Coleman

7 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95007 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database

8 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95008 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database # Database: Ayers and Chelson created a data set from a recently compiled OUSD database containing annual Selected Acquisition Reports (SARs) reported through June 1998. # The programs chosen for the study, out of the hundreds available, were narrowed down: # All programs active in 1997 # All programs were over 2 years in maturity # Eliminated programs with classified financial data # Eliminated programs with inconsistent SARs (i.e., re- baselined programs) # Eliminated programs without a summary-level variance analysis

9 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-95009 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database By Service: Army -- 13 Navy -- 17 AF -- 12 DOD -- 2 By Platform: Aircraft -- 8 Combat Vehicle -- 4 Electronic -- 11 Missile -- 8 Munitions -- 2 Ship -- 7 Space -- 4 By Estimate Type: Planning -- 20 Development -- 32 Production -- 19 (many programs spanned more than one milestone period) 435 individual SAR reports representing 44 programs dating back to 1982.

10 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950010 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database The PE--Procurement and PdE--RDT&E lacked sufficient data points for analysis. RDT&EProcurement Planning (PE)X Development (DE)XX Production (PdE)X # Programs categorized into six groups:

11 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950011 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database # Database construction: # Program Name, Service, Platform, Estimate Type, Base Year & Year of SAR # Breakdown for each SAR’s Estimating Variance: # To categorize among the “estimating” sub-classifications, used the explanations in the “Variance Analysis” section of each SAR. # For each program, pulled out Strict CE from all SARs (typically one per year), then summed to get the cumulative Strict CE Variance. # All cost elements converted to FY88 dollars

12 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950012 1999 Joint ISPA/SCEA Int’l Conf. Analysis

13 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950013 1999 Joint ISPA/SCEA Int’l Conf. Analysis # Remaining Strict (pure) CE error analyzed. # Computed first, second, and third moments # Mean # Standard deviation # Skewness # Performed tests for Distribution # Kolmogorov-Smirnov (K-S) type distribution test for normality # Performed tests for Correlation # Parametric-Pearson’s # Nonparametric-Spearman’s

14 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950014 1999 Joint ISPA/SCEA Int’l Conf. Results--Total and Strict CE By phase: DE--RDT&E and DE--Procurement were not correlated. Total CE PE RDT&E DE RDT&E DE Proc Prod Proc Mean15.4%22.3%15.0%6.0% Std Dev36.9%39.2%89.6%8.3% CV2.41.86.01.4 Skewness3.1232.6935.1691.289 0.2650.2600.4500.242 FAIL Kolmogorov- Smirnov Strict CE: Lower Means Smaller Standard Deviations Decreased Skewness Strict CE: Lower Means Smaller Standard Deviations Decreased Skewness

15 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950015 1999 Joint ISPA/SCEA Int’l Conf. Results--Means Corrected means stepped down a simple average of 8.7 percentage points (58.7% decrease)

16 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950016 1999 Joint ISPA/SCEA Int’l Conf. Results--Weighted Average RDT&EProcurement PE 7.0% DE 10.7%3.3% PdE 3.0% Weighted Avg by Phase 8.7%3.2% Cost Weight Factor 15 Total Weighted Avg 4.1% Weighted Average of Means BMDO Model currently uses 3.4%

17 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950017 1999 Joint ISPA/SCEA Int’l Conf. Interpretation--Weighted Mean # Means for RDT&E and Procurement were numerically different # RDT&E 8.7% # Procurement3.2% # Grand weighted mean4.1% Current CE factor3.4% # Means for RDT&E and Procurement did not test as statistically significantly different # But, it is intuitively attractive that earlier errors have a larger bias # RDT&E mean is larger than Procurement mean # Omissions by cost estimators more likely early on

18 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950018 1999 Joint ISPA/SCEA Int’l Conf. Histograms--Strict CE

19 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950019 1999 Joint ISPA/SCEA Int’l Conf. Kolmogorov- Smirnov Test Statistic Cumulative CE Ratio Distribution DE Procurement Example: Max Distance =.1094

20 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950020 1999 Joint ISPA/SCEA Int’l Conf. Tests for Normality (K-S) PASS FAIL PASS

21 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950021 1999 Joint ISPA/SCEA Int’l Conf. Results--Distribution # Passed K-S Test for Normality: # PE RDT&E # DE RDT&E # DE Procurement # Failed K-S Test for Normality: # PdE Procurement # The Normal distribution is a good approximation for all CE Risk # No particular reason to believe that PdE Procurement errors are non-normal # PdE errors are fairly symmetric Possible Type I Error

22 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950022 1999 Joint ISPA/SCEA Int’l Conf. Results--Correlation for Strict CE # Correlation: None # Note: only where it could be tested, between DE -- RDT&E and DE -- Procurement # Currently the BMDO model passes CE risk between phases using Phase-to-phase functional correlation # Sked/Tech Risk needs to be correlated for computational simplicity, accuracy and fidelity reasons # CE Risk is correlated for computational simplicity only. There are no adverse effects. # The Phase-to-phase functional correlation in CE will likely be left for simplicity # Some computational difficulties will be introduced if different means are used by phase, but this is not insurmountable

23 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950023 1999 Joint ISPA/SCEA Int’l Conf. Comparison to Previous Study

24 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950024 1999 Joint ISPA/SCEA Int’l Conf. Comparison of Moments All the standard deviations are comparable. The skewness picture is quite different. Conclusion: First and second moments are similar Conclusion: First and second moments are similar The new DE total mean is comparable to the old mean. But, the component means are different.

25 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950025 1999 Joint ISPA/SCEA Int’l Conf. Comparison of Distribution N = 21 Total of DE--RDT&E and DE--Proc N = 32 New Data Old Data Conclusion: Data distributions are similar Conclusion: Data distributions are similar

26 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950026 1999 Joint ISPA/SCEA Int’l Conf. Interpretation--Bias # It was formerly thought that CE Error would prove to be unbiased # CERs have no inherent bias # 3.4% is arguably indistinguishable from 0.0% # Slight negative skewness # Contradiction in mean and skewness suggested zero bias # This study strongly suggests CE Risk is biased upward  Independent data # Same average overall as before # Coherent in sub stratification -- bias and skewness consistent # Positive skewness reinforces positive bias  We conclude that positive bias is demonstrated # Perhaps this results from omissions in the initial estimate rather than a tendency for CERs to be biased # In any event, biased it is!

27 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950027 1999 Joint ISPA/SCEA Int’l Conf. Interpretation--Bias # It was formerly thought that CE Error would prove to be unbiased # CERs have no inherent bias # 3.4% is arguably indistinguishable from 0.0% # Slight negative skewness # Contradiction in mean and skewness suggested zero bias # This study strongly suggests CE Risk is biased upward # Data independently constructed # Seven additional years of data. This dataset done in 1998 v. 1991. # 24% (5 out of 21) programs repeated from the previous study to the current study. # Same average overall as before # Coherent in sub stratification -- bias and skewness consistent # Positive skewness reinforces positive bias  We conclude that positive bias is demonstrated # Perhaps this results from omissions in the initial estimate rather than a tendency for CERs to be biased # In any event, biased it is!

28 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950028 1999 Joint ISPA/SCEA Int’l Conf. Conclusions # First, second, and third moments examined # Gained insight into CE Risk by phase and maturity # First and second moments reinforced # Positive bias is strongly suggested # Normal distribution is a reasonable model # Data skewness is tolerable # Current treatment of CE in the BMDO model justified # Weighted average analysis yielded ~ 4% # Opportunity for differences by phase if desired # Cost Estimating Risk is now as well understood as Sked/Tech Risk # Rigor and accuracy are increased # Underlying philosophy is unchanged

29 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950029 1999 Joint ISPA/SCEA Int’l Conf. Back-up Slides

30 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950030 1999 Joint ISPA/SCEA Int’l Conf. Sample SAR

31 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950031 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database

32 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950032 1999 Joint ISPA/SCEA Int’l Conf. New SAR Database

33 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950033 1999 Joint ISPA/SCEA Int’l Conf. Correlation--RDT&E and Procurement Spearman’s Pearson’s

34 hfchelson@tasc.com, 703-633-8300 x4821; jayers@bmdo.mcri.com, 703-416-950034 1999 Joint ISPA/SCEA Int’l Conf. Scatter Plots-- Strict CE vs. Init Est

Download ppt "Analysis and Implementation of Cost Estimating Risk in the Ballistic Missile Defense Organization (BMDO) Risk Model A Study of Distribution Presented to."

Similar presentations

I OWA S TATE U NIVERSITY Department of Animal Science Using Basic Graphical and Statistical Procedures (Chapter in the 8 Little SAS Book) Animal Science.

I OWA S TATE U NIVERSITY Department of Animal Science Using Basic Graphical and Statistical Procedures (Chapter in the 8 Little SAS Book) Animal Science.
Statistical Techniques I EXST7005 Start here Measures of Dispersion.

Statistical Techniques I EXST7005 Start here Measures of Dispersion.
The Statistical Analysis of the Impact of Adequate Testing on the Defense Systems Development Zaw P. Tun GWU, EMSE PhD Candidate Dr. Shahram Sarkani, Ph.D.,

The Statistical Analysis of the Impact of Adequate Testing on the Defense Systems Development Zaw P. Tun GWU, EMSE PhD Candidate Dr. Shahram Sarkani, Ph.D.,
Tips for Writing Free Response Questions on the AP Statistics Exam Laura Trojan Cannon School.

Tips for Writing Free Response Questions on the AP Statistics Exam Laura Trojan Cannon School.
SCEA 2000 - 15 June 2000 JRS, TASC, 5/7/2015, 1 BMDO Cost Risk Improvement in Operations and Support (O&S) Estimates J. R. Summerville,

SCEA June 2000 JRS, TASC, 5/7/2015, 1 BMDO Cost Risk Improvement in Operations and Support (O&S) Estimates J. R. Summerville,
Simple Linear Regression. G. Baker, Department of Statistics University of South Carolina; Slide 2 Relationship Between Two Quantitative Variables If.

Simple Linear Regression. G. Baker, Department of Statistics University of South Carolina; Slide 2 Relationship Between Two Quantitative Variables If.
Correlation and regression

Correlation and regression
Research Methods in MIS: Sampling Design

Research Methods in MIS: Sampling Design
1 BA 555 Practical Business Analysis Housekeeping Review of Statistics Exploring Data Sampling Distribution of a Statistic Confidence Interval Estimation.

1 BA 555 Practical Business Analysis Housekeeping Review of Statistics Exploring Data Sampling Distribution of a Statistic Confidence Interval Estimation.
Chapter 13 Introduction to Linear Regression and Correlation Analysis

Chapter 13 Introduction to Linear Regression and Correlation Analysis
SIMPLE LINEAR REGRESSION

SIMPLE LINEAR REGRESSION
COURSE: JUST 3900 INTRODUCTORY STATISTICS FOR CRIMINAL JUSTICE Instructor: Dr. John J. Kerbs, Associate Professor Joint Ph.D. in Social Work and Sociology.

COURSE: JUST 3900 INTRODUCTORY STATISTICS FOR CRIMINAL JUSTICE Instructor: Dr. John J. Kerbs, Associate Professor Joint Ph.D. in Social Work and Sociology.
Lecture II-2: Probability Review

Lecture II-2: Probability Review
RLC, TASC, 8/8/2015, 1 (703) 631-2000 x2181 Risk In Cost Estimating General Introduction & The BMDO Approach 33 rd ADoDCAS 2-4 February.

RLC, TASC, 8/8/2015, 1 (703) x2181 Risk In Cost Estimating General Introduction & The BMDO Approach 33 rd ADoDCAS 2-4 February.
Relationships Among Variables

Relationships Among Variables
Example of Simple and Multiple Regression

Example of Simple and Multiple Regression
The Data Analysis Plan. The Overall Data Analysis Plan Purpose: To tell a story. To construct a coherent narrative that explains findings, argues against.

The Data Analysis Plan. The Overall Data Analysis Plan Purpose: To tell a story. To construct a coherent narrative that explains findings, argues against.
Math 116 Chapter 12.

Math 116 Chapter 12.
Developed by© 2002 SCEA. All rights reserved. CostPROF Unit III - Module 91 Cost Risk Analysis How to adjust your estimate for historical cost growth.

Developed by© 2002 SCEA. All rights reserved. CostPROF Unit III - Module 91 Cost Risk Analysis How to adjust your estimate for historical cost growth.
HAWKES LEARNING SYSTEMS math courseware specialists Copyright © 2010 by Hawkes Learning Systems/Quant Systems, Inc. All rights reserved. Chapter 14 Analysis.

HAWKES LEARNING SYSTEMS math courseware specialists Copyright © 2010 by Hawkes Learning Systems/Quant Systems, Inc. All rights reserved. Chapter 14 Analysis.

Similar presentations

Ads by Google