1. Systems Engineering Cost Estimation Systems Engineering Day, São José dos Campos, Brazil Dr. Ricardo Valerdi Massachusetts Institute of Technology June 6, 2011 [rvalerdi@mit.edu]

2. Theory is when you know everything, but nothing works. Practice is when everything works, but no one knows why. Harvard is where theory and practice come together... Nothing works and no one knows why. - on the door of a laboratory at Harvard

3. The Delphic Sybil Michelangelo Buonarroti Capella Sistina, Il Vaticano (1508-1512)

4. 4 Cost Commitment on Projects Blanchard, B., Fabrycky, W., Systems Engineering & Analysis, Prentice Hall, 1998.

5. 5 FeasibilityPlans/Rqts.DesignDevelop and Test Phases and Milestones Relative Size Range Operational Concept Life Cycle Objectives Life Cycle Architecture Initial Operating Capability x 0.5x 0.25x 4x 2x Cone of Uncertainty Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.

6. 6 How is Systems Engineering Defined? Acquisition and Supply –Supply Process –Acquisition Process Technical Management –Planning Process –Assessment Process –Control Process System Design –Requirements Definition Process –Solution Definition Process Product Realization –Implementation Process –Transition to Use Process Technical Evaluation –Systems Analysis Process –Requirements Validation Process –System Verification Process –End Products Validation Process EIA/ANSI 632, Processes for Engineering a System, 1999.

7. COSYSMO Data Sources BoeingIntegrated Defense Systems (Seal Beach, CA) RaytheonIntelligence & Information Systems (Garland, TX) Northrop GrummanMission Systems (Redondo Beach, CA) Lockheed MartinTransportation & Security Solutions (Rockville, MD) Integrated Systems & Solutions (Valley Forge, PA) Systems Integration (Owego, NY) Aeronautics (Marietta, GA) Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY) General DynamicsMaritime Digital Systems/AIS (Pittsfield, MA) Surveillance & Reconnaissance Systems/AIS (Bloomington, MN) BAE Systems National Security Solutions/ISS (San Diego, CA) Information & Electronic Warfare Systems (Nashua, NH) SAIC Army Transformation (Orlando, FL) Integrated Data Solutions & Analysis (McLean, VA) L-3 Communications Greenville, TX

8. 8 COSYSMO Scope Addresses first four phases of the system engineering lifecycle (per ISO/IEC 15288) Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632) Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

9. 9 COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms + 3 Adj. Factors - Application factors -8 factors - Team factors -6 factors COSYSMO Operational Concept

10. 10 COSYSMO Model Form Where: PM NS = effort in Person Months (Nominal Schedule) A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN} w x = weight for “easy”, “nominal”, or “difficult” size driver = quantity of “k” size driver E = represents diseconomies of scale EM = effort multiplier for the j th cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort.

11. 11 UNDERSTANDING FACTORS –Requirements understanding –Architecture understanding –Stakeholder team cohesion –Personnel experience/continuity COMPLEXITY FACTORS –Level of service requirements –Technology Risk –# of Recursive Levels in the Design –Documentation Match to Life Cycle Needs OPERATIONS FACTORS –# and Diversity of Installations/Platforms –Migration complexity PEOPLE FACTORS –Personnel/team capability –Process capability ENVIRONMENT FACTORS –Multisite coordination –Tool support Cost Driver Clusters

12. 12 Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team. 1.51.221.000.810.65 ViewpointVery LowLowNominalHighVery High Culture  Stakeholders with diverse expertise, task nature, language, culture, infrastructure  Highly heterogeneous stakeholder communities  Heterogeneous stakeholder community  Some similarities in language and culture  Shared project culture  Strong team cohesion and project culture  Multiple similarities in language and expertise  Virtually homogeneous stakeholder communities  Institutionalized project culture Compatibility  Highly conflicting organizational objectives  Converging organizational objectives  Compatible organizational objectives  Clear roles & responsibilities  Strong mutual advantage to collaboration Familiarity and trust  Lack of trust  Willing to collaborate, little experience  Some familiarity and trust  Extensive successful collaboration  Very high level of familiarity and trust

13. Technology Risk The maturity, readiness, and obsolescence of the technology being implemented. Immature or obsolescent technology will require more Systems Engineering effort. ViewpointVery LowLowNominalHighVery High Lack of Maturity Technology proven and widely used throughout industry Proven through actual use and ready for widespread adoption Proven on pilot projects and ready to roll-out for production jobs Ready for pilot useStill in the laboratory Lack of Readiness Mission proven (TRL 9) Concept qualified (TRL 8) Concept has been demonstrated (TRL 7) Proof of concept validated (TRL 5 & 6) Concept defined (TRL 3 & 4) Obsolescen ce - Technology is the state-of-the- practice - Emerging technology could compete in future - Technology is stale - New and better technology is on the horizon in the near-term - Technology is outdated and use should be avoided in new systems - Spare parts supply is scarce

14. Migration complexity This cost driver rates the extent to which the legacy system affects the migration complexity, if any. Legacy system components, databases, workflows, environments, etc., may affect the new system implementation due to new technology introductions, planned upgrades, increased performance, business process reengineering, etc. ViewpointNominalHighVery HighExtra High Legacy contractor Self; legacy system is well documented. Original team largely available Self; original development team not available; most documentation available Different contractor; limited documentation Original contractor out of business; no documentation available Effect of legacy system on new system Everything is new; legacy system is completely replaced or non-existent Migration is restricted to integration only Migration is related to integration and development Migration is related to integration, development, architecture and design

15. 15 Cost Driver Rating Scales Very LowLowNominalHighVery High Extra HighEMR Requirements Understanding1.871.371.000.770.60 3.12 Architecture Understanding1.641.281.000.810.65 2.52 Level of Service Requirements0.620.791.001.361.85 2.98 Migration Complexity 1.001.251.551.93 Technology Risk0.670.821.001.321.75 2.61 Documentation0.780.881.001.131.28 1.64 # and diversity of installations/platforms 1.001.231.521.87 # of recursive levels in the design0.760.871.001.211.47 1.93 Stakeholder team cohesion1.501.221.000.810.65 2.31 Personnel/team capability1.501.221.000.810.65 2.31 Personnel experience/continuity1.481.221.000.820.67 2.21 Process capability1.471.211.000.880.770.682.16 Multisite coordination1.391.181.000.900.800.721.93 Tool support1.391.181.000.850.72 1.93

16. 16 Cost Drivers Ordered by Effort Multiplier Ratio (EMR)

17. ISO/IEC 15288 Conceptualize Develop Transition to Operation Acquisition & Supply Technical Management System Design Product Realization Technical Evaluation Operational Test & Evaluation ANSI/EIA 632 Effort Profiling

18. 18 Before Local Calibration

19. 19 After Local Calibration

20. 20 Prediction Accuracy PRED(30) PRED(25) PRED(20) PRED(30) = 100% PRED(25) = 57%

21. 21 Academic prototype Commercial Implementations Proprietary Implementations COSYSMO-R SECOST SEEMaP Impact Academic Curricula Intelligence Community Sheppard Mullin, LLC Policy & Contracts Model 10 theses

22. 22 Contact Ricardo Valerdi MIT rvalerdi@mit.edu (617) 253-8583 http://rvalerdi.mit.edu