1. The Statistical Analysis of the Impact of Adequate Testing on the Defense Systems Development Zaw P. Tun GWU, EMSE PhD Candidate Dr. Shahram Sarkani, Ph.D., P.E. Dr. Thomas Mazzuchi, D.Sc. May 1, 2015

2. Overview  Purpose  Background  Focus Area  Data Extraction  Source of Data  Proposed Methodology  Conclusion  Future Work

3. Introduction  Purpose Analyze the cost growth trend to determine if adequate testing plays an essential role in successful development of new systems in the DOD acquisition environment  Background DOD is managing $1.68 trillion worth of acquisition programs DOD’s acquisition costs has increased by $135 billion over the past 2 years for 98 major defense acquisition programs GAO found that most of DOD’s portfolio total cost growth in past 2 years occurred in the DOD’s largest programs, which are all in production –This means programs experience significant problems and changes well after the programs and their costs should have stabilized

4. Focus Areas  GAO has continuously stated that in order to mitigate risks of uncovering problems in the later phases of the development cycle, knowledge based testing should be conducted at key junctures, specifically around the Engineering and Manufacturing Development (EMD) phase Source: GAO (2011), Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP

5. Data Extraction Mission Need Statement Between MS-B and MS-C Appropriate TRL level (GAO knowledge point 1) PDR (GAO knowledge point 1) CDR (GAO knowledge point 2) Minimum design changes after this point Manufacturing process should be stabilized before entering Production phase (GAO knowledge point 3) Should not have significant cost growth at this point Beyond MS-C RDT&E cost growth should be minimum Data form SAR, GAO, and DOT&E Annual Report, etc. Significant RDT&E cost growth in the Production and Development Phase is the indicator of insufficient T&E activities in earlier phases of the acquisition

6. Primary Source of Data  Selected Acquisition Reports (SARs) RAND conducted several research studies based on SAR data and identified numerous “disadvantages" that an analyst must be aware of in order to maintain the validity of the data. Most prevalent problems are: o Failure of some programs to use a consistent baseline cost estimate o Exclusion of some significant elements of cost o Exclusion of certain classes of major programs (e.g., special access programs) o Constantly changing preparation guidelines o Inconsistent interpretation of preparation guidelines across programs o Unknown and variable funding levels for program risk o Cost sharing in joint programs o Reporting of effects of cost changes rather than their root causes Using SARs provides key “advantages” o SARs conform to a strict reporting format, providing consistency to the data o Those who create SAR reports receive annual SAR training, which adds to the consistency of the data o SARs are presented to Congress, so the level of scrutiny that SARs receive in the review process bolsters both the consistency and accuracy of the documents. Databases in general contain inaccuracies, but a database built from SAR data arguably withstands scrutiny better than most

7. Supplementary Data Sources  DOT&E Annual Reports Current Activities Cost data Issues Milestone dates Recommendations  GAO Reports Technology, Design, and Production Maturity Cost Data Milestone Data Issues

8. Conceptual Model Performance Indicators

9. Proposed Methodology  Data Extraction Base year RDT&E dollars from SAR chosen for analysis, since these dollars exclude estimated inflationary effects This format facilitates conversion of the various base years of individual estimates into a single base year, making possible easy comparison across programs  Data Analysis Relationship between early adequate testing and program success –RDT&E cost growth Prior/Post MS-C  Schedule Slip and Unit Cost Growth –Performance indicators  Schedule Slip and Unit Cost Growth

10. Using SEM for Data Analysis  Powerful multivariate techniques – Specialized version of other analysis methods  Advantages over Multiple Regression SEM programs provide overall tests of model fit and individual parameter estimate tests simultaneously Regression coefficients, means, and variances may be compared simultaneously, even across multiple between subjects groups Measurement and confirmatory factor analysis models can be used to purge errors, making estimated relationships among latent variables less contaminated by measurement error Ability to fit non-standard models, including flexible handling of longitudinal data, databases with auto correlated error structures (time series analysis), and databases with non-normally distributed variables and incomplete data

11. An Example of Cost Growth Due to Inadequate Early Testing Compare Baseline and Current Status(Cost and Schedule) RDT&E Cost increase in Production Phase indicate that more problems discovers during that phase It is an indicator that program did not perform adequate early testing GAO Early Knowledge Based Testing Activities

12.  Recently DOT&E conducted a systematic review of recent 67major acquisition programs that experienced delays 56 programs (or 84 percent) had performance problems in testing (either DT, OT, or both)  56 programs were categorized and cost growth were analyzed and compared with other programs in the same category  Preliminary results shows that programs with inadequate testing in the earlier phase of the development has higher cost growth Preliminary Results Average % Growthn Aircraft 171.65 16 Helicopter139.898 Ship116.948 Missile95.3511 Satellite90.929 Land Vehicle84.826 C3I42.1213 Inadequate Testing Cost Growth Other Cost Growth Aircraft244.94%114.64% Missile116.26%77.92%  Preliminary data analysis shows that aircraft acquisition programs has highest average percentage of total acquisition cost growth compared to others programs.

13. Conclusion  Knowledge based testing is essential in detecting problems early in the acquisition phase  Program success depends on the knowledge obtained during earlier tests in the development of the system  Structural Equation Modeling (SEM) is a powerful statistical technique to analyze program success factors from the T&E perspective.

14. Future Work  Complete Data collection and data base construction  Refine Performance Indicators and Conceptual Model based on further research  Perform data analysis using Structural Equation Modeling (SEM) to test model and find the correlation between program success and “Performance Indicators”

15. Contact Information Zaw P. Tun (407) 595-4684 zph2k@gwmail.gwu.edu Shahram Sarkani, Ph.D., P.E (888) 694-9627 sarkani@gwu.edu Thomas A. Mazzuchi, D.Sc. (202) 994-7541 mazzu@gwu.edu

16. Backup

17.  References Arena, M., Leonard, R., Murray, S., & Younossi, O. (2006). Historical Cost Growth of Completed Weapon System Programs. Santa Monica CA: RAND GAO (2011). Defense Acquisition: Assessments of Selected Weapon Programs, GAO-11-233SP GAO (2011). WEAPONS ACQUISITION REFORM :Actions Needed to Address Systems Engineering and Developmental Testing Challenges, GAO-11-806 Hough, Paul G. (1992). Pitfalls in Calculating Cost Growth from Selected Acquisition Reports. Santa Monica CA: RAND Jarvaise, J., Drenzer, J., & Norton, D. (1996). The Defense System Cost Performance Database: Cost Growth Analysis Using Selected Acquisition Reports. Santa Monica CA: RAND Sipple, V. (2002). ESTIMATING ENGINEERING COST RISK USING LOGISTIC AND MULTIPLE REGRESSION, Master Thesis, AFIT. The office of the Director, Operational Test and Evaluation (DOT&E) (2011). FY 2011 Annual Report