Example © 2012 Lockheed Martin Corporation. All Rights Reserved. October 2012 Proxy Estimation Costing for Systems (PECS) Reggie Cole Lockheed Martin Senior Fellow

2 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Discussion Topics  Why Do We Need Yet Another Cost Model? –The gap in early-stage system cost modeling  Systems Engineering Effort as a Proxy Estimator for System Cost –And the role of COSYSMO is arriving at this proxy estimate  Proxy Estimation Costing for Systems (PECS) –Derivation of the PECS Model –The PECS modeling approach  Case Study for Affordability Analysis Using the PECS Model –The real power of the PECS model

3 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy – So Does Solution Information Problem-Space Description Cost Estimate ± 25% High-Level Solution Description Cost Estimate ± 10% Detailed Solution Description Cost Estimate ± 5% High-Level Solution Assumptions Cost Estimate ± 20% Increasing Effort and Cost-Modeling Expertise Increasingly Refined Information About the Solution Increasingly Refined Cost Estimate Increasingly Refined Solution We Have a Good Selection of Tools for Late-Stage Cost Modeling We Have Gaps in Early-Stage Cost Modeling

4 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Systems Engineering Effort as a Proxy Measure of Overall System Size and Complexity  Proxy Measures –Proxy measures are used when you cannot directly measure what you want to measure – and when an indirect measure provides sufficient insight –Proxy measures are often used in clinical studies since direct measurement is often infeasible or can even alter the outcome –It is not always possible to directly measure what you want to measure – or directly estimate what you want to estimate  System Engineering Effort is a Proxy Measure for System Cost –There is strong evidence for the link between systems engineering effort and program cost – dating back to a NASA study in the 1980s –The optimal relationship between systems engineering effort and overall program cost is 10% - 15% –Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems –Systems engineering effort can be an effective proxy measure for overall system cost H. Dickinson, S. Hrisos, M. Eccles, J. Francis, M. Johnston, Statistical Considerations in a Systematic Review of Proxy Measures of Clinical Behaviour, Implementation Science, 2010 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

5 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size, Complexity and Reuse Number of System Requirements Number of Major System Interfaces Number of Critical Algorithms Number of Operational Scenarios Size Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Cost Drivers Managed Elements Adopted Elements Deleted Elements Modified Elements New Elements Reuse Factors Initial Estimate of System Size Scaled Estimate of System Size Consolidated Cost Driver Factor Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost

6 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. An Approach for De-Biasing the Proxy Estimator – Relationship Between SE Effort and Total Effort NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004 INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort

7 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. The PECS Cost Function VariableTypeDescription COSYSMO Calibration FactorDeterministic Scalar ValueOrganization-specific calibration factor Effort Conversion FactorTriangular Distributed Random VariableThree-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16) SE EffortTriangular Distributed Random VariableThree-point estimate for SE effort, generated using COSYSMO Labor RateTriangular Distributed Random VariableThree-point estimate for composite labor rate Material CostsTriangular Distributed Random VariableThree-point estimate for material costs Travel CostsTriangular Distributed Random VariableThree-point estimate for travel costs This Model is Well Positioned for Monte Carlo Analysis

8 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. The PECS Model – Putting It All Together  Size Drivers (Problem Space)  Customer Requirements  System Interfaces  Major Algorithms  Operational Scenarios  Complexity Drivers (Problem/Solution)  Requirements Understanding  Architecture Understanding  Level of Service Requirements  Migration Complexity  Technology Risk  Documentation Needs  Installations/Platform Diversity  Levels of Recursion in the Design  Stakeholder Team Cohesion  Personnel/Team Capability  Personnel Experience/Continuity  Process Capability  Multisite Coordination  Tool Support  Reuse Factors (Solution Space)  New  Modified  Deleted  Adopted  Managed SE Effort is an estimator for total system cost…but it is a biased estimator Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort Proxy Estimation Costing for Systems (PECS) Estimator De-Biasing Monte Carlo Analysis of System Cost Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for the Monte Carlo analysis of system cost

9 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Case Study – The COSYSMO Scenarios The case study is based on a large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers. The case study looks at three cost scenarios:  Case 1 – The original unprecedented system (for calibration purposes)  Case 2 – Replacement system (as a new development)  Case 3 – Replacement system (as a largely COTS/GOTS approach) COSYSMO Scenarios for PECS – Three Scenarios for Each Case

10 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Case Study – The Monte Carlo Analysis Case 1 Average 80/20 Cost = $1.9B Used as a calibration point for the model Case 2 Average 80/20 Cost = $77M Initial Solution for Replacement System Case 3 Average 80/20 Cost = $30M More Affordable Solution, Based on COTS/GOTS Solution The PECS Model Enables Rapid Turn-Around Analysis of Alternatives and “Should Cost” Analysis

11 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Conclusion  The PECS Model is Based on Well-Established Approaches –COSYSMO provides the basis for estimation of systems engineering effort – and a biased proxy estimator for overall system cost –There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort –Monte Carlo analysis is a well-established technique for cost modeling  The PECS Model Can Improve System Cost Modeling –The PECS Model occupies an important niche – fully parametric system cost modeling in the early stages of system definition –The PECS Model can serve as a powerful affordability analysis tool – supporting rapid-turnaround analysis of alternatives –But…the PECS Model is not a replacement for existing models  Next Steps –Broader validation of the model –Cross-industry review of the model

12 Example © 2012 Lockheed Martin Corporation. All Rights Reserved.