1. March 2002 COSYSMO: COnstructive SYStems Engineering Cost MOdel Ricardo Valerdi USC Annual Research Review March 11, 2002

2. March 20022 Outline Background on COSYSMO EIA632 Approach Delphi Survey Delphi Round 1 Results Analysis/Conclusions Lessons Learned/Improvements The Next Step Q & A

3. March 20023 “All models are wrong, but some of them are useful” - W. E. Deming Source: www.deming.org

4. March 20024 What is it? The purpose of the COSYSMO project is to enhance the current capability of the COCOMO II model by accounting for costs that are outside the realm of software engineering by introducing system engineering drivers.

5. March 20025 The Challenge To develop a preliminary model for estimating the cost impact of front- end System Engineering tasks in the design of software intensive systems. These include system definition, integration, and test activities as defined in standard EIA632.

6. March 20026 Approach Begin with front-end costs of information systems engineering Follow 7-step modeling methodology –Steps 5,6, & 7 (gather data and refine loop) Use model parameters compiled by TRW, SAIC, Raytheon, and USC/CSE –System size (requirements, TPMs, I/F) –Effort drivers (maturity, cohesion, stability)

7. March 20027 COCOMO Suite COCOMOII COQUALMO COPSEMO CORADMO COSYSMO COPROMO COCOTS

8. March 20028 COSYSMO Operational Concept COSYSMO Size Drivers Cost Drivers Effort Duration Calibration # Requirements # Interfaces # TPM’s # Scenarios # Modes # Platforms # Algorithms - 7 Application factors - 8 Team factors WBS guided By EIA 632 COCOMO II-based model

9. March 20029 EIA632 33 activities organized into 5 groups: 1.Acquisition and supply 2.Technical management Planning process Assessment process Control process 3.System design Requirements definition process Solution definition process 4.Product realization 5.Technical evaluation

10. March 200210 Delphi Survey 3 Sections: –Scope, Size, Cost Used to determine the range for size driver and effort multiplier ratings Identify the cost drivers to which effort is most sensitive to Reach consensus from systems engineering experts

11. March 200211 7 Size Drivers 1. Number of System Requirements 2. Number of Major Interfaces 3. Number of Technical Performance Measures 4. Number of Operational Scenarios 5. Number of Modes of Operation 6. Number of Different Platforms 7. Number of Unique Algorithms

12. March 200212 15 Cost Drivers 1. Requirements understanding 2. Architecture understanding 3. Level of service requirements, criticality, difficulty 4. Legacy transition complexity 5. COTS assessment complexity 6. Platform difficulty 7. Required business process reengineering Application Factors (7)

13. March 200213 15 Cost Drivers (cont…) 1. Number and diversity of stakeholder communities 2. Stakeholder team cohesion 3. Personnel capability 4. Personal experience/continuity 5. Process maturity 6. Multisite coordination 7. Formality of deliverables 8. Tool support Team Factors (8)

14. March 200214 Delphi Round 1 23 Surveys returned Aerospace2 Galorath1 Lockheed Martin8 Raytheon4 SAIC6 TRW1 USC1

15. March 200215 System Engineering Effort Per EIA Stage Stage Supplier Performance Technical Management Requirements Definition Solution Definition Systems Analysis Requirements Validation Design Solution Verification End Products Validation Delphi 5.2% 13.1% 16.6% 18.1% 19.2% 11.3% 10.5% 6.6% Suggested 5% 15% 20% 15% 5% 3.05 4.25 4.54 4.28 5.97 4.58 6.07 3.58 Std. Dev.

16. March 200216 Delphi Round 1 Highlights (cont.) Range of sensitivity for Size Drivers # Algorithms # Requirements # Interfaces # TPM’s # Scenarios # Modes # Platforms 5.57 Relative Effort 1 2.23 2.54 2.212.10 6.48 6 4 2

17. March 200217 Two Most Sensitive Size Drivers Suggested Rel. Effort Delphi Respondents EMR Rel. Effort Standard Deviation # Interfaces 45.571.80 # Algorithms 66.482.09

18. March 200218 Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Application Factors) EMR 1.93 2.81 2.13 2.43 2.24 4 2 Requirements und. Architecture und. Level of service reqs. Legacy transition COTS Platform difficulty Bus. process reeng. 1.741.13

19. March 200219 Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Team Factors) 1.28 2.46 1.91 2.16 1.94 1.25 Tool support Stakeholder comm. Stakeholder cohesion Personnel capability Personal experience Process maturity Multisite coord. Formality of deliv. 1.841.78 EMR 4 2

20. March 200220 Suggested EMR Delphi Respondents EMR Mean Standard Deviation Arch. Under. 1.662.240.83 Reqs. Under. 1.732.430.70 Pers. Cap. 2.152.460.66 Serv. Req. 2.52.810.67 Four Most Sensitive Cost Drivers

21. March 200221 Conclusions Not only do we need to better manage requirements, we also need to manage: 1) # of Interfaces 2) # of Algorithms 3) Personnel Capability 4) Level of service requirements, criticality, difficulty 5) Level of understanding “Control the controllables”

22. March 200222 Lessons Learned/Improvements Lesson 1 – There are lots of people and groups Interested in more precisely estimating system costs - And they are willing to help do it free. Lesson 2 – Currently, system engineering effort is estimated using activity-based costing heuristics Lesson 3 – When mounting a Delphi, clearly identify what you are trying to do. - Else, system engineers will attack you with a shotgun Lesson 4 – We could use help in developing a better designed template for Delphi instruments

23. March 200223 The Next Steps Incorporate suggestions from Delphi 1 Write report Get Masters degree Data from completed systems will then be used to statistically confirm or deny initial ratings Round 2 of Delphi …in the future

24. March 200224 Special Thanks to: Advisors Dr. Boehm, Dr. Axelband, Don Reifer Affiliates Gary Hafen, Tony Jordano, Chris Miller, Karen Owens, Don Reifer, Garry Roedler, Evin Stump, Gary Thomas, Marilee Wheaton

25. March 200225 Where can I get more info? COSYSMO Working Group meeting Thursday March 14 th 12:00 - 5:00 valerdi.com/cosysmo

26. March 200226

27. March 200227 Delphi Round 1 Participants