Analysis and Implementation of Cost Estimating Risk in the Ballistic Missile Defense Organization (BMDO) Risk Model A Study of Distribution Presented to the 1999 Joint ISPA/SCEA International Conference 8-11 June 1999 Prepared for: Deputy for Program Operations THE BALLISTIC MISSILE DEFENSE ORGANIZATION 7100 Defense Pentagon Washington, D.C Prepared by: Heather F. Chelson, TASC, Inc. Jessica R. Ayers, TASC, Inc. Richard L. Coleman, TASC, Inc. Nadiyeh Shahidi, TASC, Inc. Donna M. Snead, BMDO x

x4821; Joint ISPA/SCEA Int’l Conf. Outline # Risk Definitions # Purpose # Approach # New SAR Database # Analysis # First, Second, and Third Moments # Distribution # Correlation # Comparison to previous study # Moments and Distribution # Interpretation of Bias # Conclusion

x4821; Joint ISPA/SCEA Int’l Conf. Risk Definitions # Cost Growth = Cost Estimating Growth + Sked/Tech Growth + Requirements Growth + Threat Growth # Cost Risk = Cost Estimating Risk + Sked/Technical Risk + Requirements Risk + Threat Risk # Cost Estimating Risk: Risk due to cost estimating errors, and the statistical uncertainty in the estimate # Schedule/Technical Risk: Risk due to inability to conquer problems posed by the intended design in the current CARD # Requirements Risk: Risk due to as-yet-unseen design shift from the current CARD arising due to CARD shortfalls # Risk arising due to the inability of the intended design to perform the (unchanged) intended mission # “Not understanding the solution” # Threat Risk: Due to as-yet-unrevealed threat shift from the current STAR # “Not understanding the problem”

x4821; Joint ISPA/SCEA Int’l Conf. Example Cost Estimate with Risk Initial Point Estimate Add Cost Estimating Risk Add Sched/Tech Risk S/T Risk CE Risk Init Pt Est $ 3% 20% 23%

x4821; Joint ISPA/SCEA Int’l Conf. Purpose # To review the Cost Estimating (CE) risk component of BMDO’s Cost Risk Methodology # The BMDO risk model currently performs risk estimation for 12 major BMDO elements including, e.g., PAC-3, THAAD, Navy Theater- Wide TBMD, and elements of the National Missile Defense Program # To gain insight into CE Risk # Large dollars (1/7 th of total risk), but small insight # CE risk is one element of BMDO’s total cost risk. The current model uses # CE risk ~ 3.4% # Schedule/technical risk ~ 20% # This results in a total risk of ~ 23%. 1 Each of these factors is passed through the model with functional correlation. 2 1 Distribution and Correlation of Sources of Cost Risk, G.E. Hartigan, J.R. Ayers, and R.L. Coleman, May An Overview of Correlation and Functional Dependencies in Cost Risk and Uncertainty Analysis, R. L. Coleman, S. S. Gupta, DoDCAS 1994, and Cost Risk Estimates Incorporating Functional Correlation, Acquisition Phase Relationships, and Realized Risk, by R.L. Coleman, S.S. Gupta, J.R. Ayers, and G.E. Hartigan, May

x4821; Joint ISPA/SCEA Int’l Conf. Approach # Update to the previous analysis 1 of BMDO’s Cost Estimating risk. To review: # Found that CE and S/T risk were uncorrelated # Internal variance, a proxy for CE risk was normally distributed # Update the model with new findings # Current model reflects the above research: Cost Estimating Risk 3.4% mean, 14.6% St Dev CE Risk is Normally distributed CE Risk is constant by phase CE Risk is computed once, and passed forward using Functional Correlation # Create new SAR database: increase size and stratification 1 Distribution and Correlation of Sources of Cost Risk, 15 May 1997, by Hartigan, Ayers and Coleman

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database # Database: Ayers and Chelson created a data set from a recently compiled OUSD database containing annual Selected Acquisition Reports (SARs) reported through June # The programs chosen for the study, out of the hundreds available, were narrowed down: # All programs active in 1997 # All programs were over 2 years in maturity # Eliminated programs with classified financial data # Eliminated programs with inconsistent SARs (i.e., re- baselined programs) # Eliminated programs without a summary-level variance analysis

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database By Service: Army Navy AF DOD -- 2 By Platform: Aircraft -- 8 Combat Vehicle -- 4 Electronic Missile -- 8 Munitions -- 2 Ship -- 7 Space -- 4 By Estimate Type: Planning Development Production (many programs spanned more than one milestone period) 435 individual SAR reports representing 44 programs dating back to 1982.

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database The PE--Procurement and PdE--RDT&E lacked sufficient data points for analysis. RDT&EProcurement Planning (PE)X Development (DE)XX Production (PdE)X # Programs categorized into six groups:

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database # Database construction: # Program Name, Service, Platform, Estimate Type, Base Year & Year of SAR # Breakdown for each SAR’s Estimating Variance: # To categorize among the “estimating” sub-classifications, used the explanations in the “Variance Analysis” section of each SAR. # For each program, pulled out Strict CE from all SARs (typically one per year), then summed to get the cumulative Strict CE Variance. # All cost elements converted to FY88 dollars

x4821; Joint ISPA/SCEA Int’l Conf. Analysis

x4821; Joint ISPA/SCEA Int’l Conf. Analysis # Remaining Strict (pure) CE error analyzed. # Computed first, second, and third moments # Mean # Standard deviation # Skewness # Performed tests for Distribution # Kolmogorov-Smirnov (K-S) type distribution test for normality # Performed tests for Correlation # Parametric-Pearson’s # Nonparametric-Spearman’s

x4821; Joint ISPA/SCEA Int’l Conf. Results--Total and Strict CE By phase: DE--RDT&E and DE--Procurement were not correlated. Total CE PE RDT&E DE RDT&E DE Proc Prod Proc Mean15.4%22.3%15.0%6.0% Std Dev36.9%39.2%89.6%8.3% CV Skewness FAIL Kolmogorov- Smirnov Strict CE: Lower Means Smaller Standard Deviations Decreased Skewness Strict CE: Lower Means Smaller Standard Deviations Decreased Skewness

x4821; Joint ISPA/SCEA Int’l Conf. Results--Means Corrected means stepped down a simple average of 8.7 percentage points (58.7% decrease)

x4821; Joint ISPA/SCEA Int’l Conf. Results--Weighted Average RDT&EProcurement PE 7.0% DE 10.7%3.3% PdE 3.0% Weighted Avg by Phase 8.7%3.2% Cost Weight Factor 15 Total Weighted Avg 4.1% Weighted Average of Means BMDO Model currently uses 3.4%

x4821; Joint ISPA/SCEA Int’l Conf. Interpretation--Weighted Mean # Means for RDT&E and Procurement were numerically different # RDT&E 8.7% # Procurement3.2% # Grand weighted mean4.1% Current CE factor3.4% # Means for RDT&E and Procurement did not test as statistically significantly different # But, it is intuitively attractive that earlier errors have a larger bias # RDT&E mean is larger than Procurement mean # Omissions by cost estimators more likely early on

x4821; Joint ISPA/SCEA Int’l Conf. Histograms--Strict CE

x4821; Joint ISPA/SCEA Int’l Conf. Kolmogorov- Smirnov Test Statistic Cumulative CE Ratio Distribution DE Procurement Example: Max Distance =.1094

x4821; Joint ISPA/SCEA Int’l Conf. Tests for Normality (K-S) PASS FAIL PASS

x4821; Joint ISPA/SCEA Int’l Conf. Results--Distribution # Passed K-S Test for Normality: # PE RDT&E # DE RDT&E # DE Procurement # Failed K-S Test for Normality: # PdE Procurement # The Normal distribution is a good approximation for all CE Risk # No particular reason to believe that PdE Procurement errors are non-normal # PdE errors are fairly symmetric Possible Type I Error

x4821; Joint ISPA/SCEA Int’l Conf. Results--Correlation for Strict CE # Correlation: None # Note: only where it could be tested, between DE -- RDT&E and DE -- Procurement # Currently the BMDO model passes CE risk between phases using Phase-to-phase functional correlation # Sked/Tech Risk needs to be correlated for computational simplicity, accuracy and fidelity reasons # CE Risk is correlated for computational simplicity only. There are no adverse effects. # The Phase-to-phase functional correlation in CE will likely be left for simplicity # Some computational difficulties will be introduced if different means are used by phase, but this is not insurmountable

x4821; Joint ISPA/SCEA Int’l Conf. Comparison to Previous Study

x4821; Joint ISPA/SCEA Int’l Conf. Comparison of Moments All the standard deviations are comparable. The skewness picture is quite different. Conclusion: First and second moments are similar Conclusion: First and second moments are similar The new DE total mean is comparable to the old mean. But, the component means are different.

x4821; Joint ISPA/SCEA Int’l Conf. Comparison of Distribution N = 21 Total of DE--RDT&E and DE--Proc N = 32 New Data Old Data Conclusion: Data distributions are similar Conclusion: Data distributions are similar

x4821; Joint ISPA/SCEA Int’l Conf. Interpretation--Bias # It was formerly thought that CE Error would prove to be unbiased # CERs have no inherent bias # 3.4% is arguably indistinguishable from 0.0% # Slight negative skewness # Contradiction in mean and skewness suggested zero bias # This study strongly suggests CE Risk is biased upward  Independent data # Same average overall as before # Coherent in sub stratification -- bias and skewness consistent # Positive skewness reinforces positive bias  We conclude that positive bias is demonstrated # Perhaps this results from omissions in the initial estimate rather than a tendency for CERs to be biased # In any event, biased it is!

x4821; Joint ISPA/SCEA Int’l Conf. Interpretation--Bias # It was formerly thought that CE Error would prove to be unbiased # CERs have no inherent bias # 3.4% is arguably indistinguishable from 0.0% # Slight negative skewness # Contradiction in mean and skewness suggested zero bias # This study strongly suggests CE Risk is biased upward # Data independently constructed # Seven additional years of data. This dataset done in 1998 v # 24% (5 out of 21) programs repeated from the previous study to the current study. # Same average overall as before # Coherent in sub stratification -- bias and skewness consistent # Positive skewness reinforces positive bias  We conclude that positive bias is demonstrated # Perhaps this results from omissions in the initial estimate rather than a tendency for CERs to be biased # In any event, biased it is!

x4821; Joint ISPA/SCEA Int’l Conf. Conclusions # First, second, and third moments examined # Gained insight into CE Risk by phase and maturity # First and second moments reinforced # Positive bias is strongly suggested # Normal distribution is a reasonable model # Data skewness is tolerable # Current treatment of CE in the BMDO model justified # Weighted average analysis yielded ~ 4% # Opportunity for differences by phase if desired # Cost Estimating Risk is now as well understood as Sked/Tech Risk # Rigor and accuracy are increased # Underlying philosophy is unchanged

x4821; Joint ISPA/SCEA Int’l Conf. Back-up Slides

x4821; Joint ISPA/SCEA Int’l Conf. Sample SAR

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database

x4821; Joint ISPA/SCEA Int’l Conf. New SAR Database

x4821; Joint ISPA/SCEA Int’l Conf. Correlation--RDT&E and Procurement Spearman’s Pearson’s

x4821; Joint ISPA/SCEA Int’l Conf. Scatter Plots-- Strict CE vs. Init Est