1. Towards COSYSMO 2.0 Future Directions and Priorities CSSE Annual Research Review Los Angeles, CA March 17, 2008 Garry Roedler Gan Wang Jared Fortune Ricardo Valerdi

2. Agenda Context setting Discussion on COSYSMO 2.0 improvements Prioritization exercise

3. 8:15 – 9:00 am  Introductions  overview of the model  summary of COSYSMO 2.0 improvements Garry Roedler 9:00 – 10:00 am  Reuse (overview only)  Integration between SwE & SysE Jared Fortune, Gan Wang 10:00 – 10:30 amBreak 10:30 – 11:00 am  Assumption of linearity in cost drivers  Cost drivers vs. scale factors Gan Wang 11:00 – 12:00 pm  Recursive levels in the design  Risk modeling (overview only) Ricardo Valerdi, Garry Roedler 12:00 – 1:00 pmLunch 1:00 – 1:45 pm  Best practice guidance Garry Roedler 1:45 – 2:30 pm  Modeling organizational factors in space systems Darryl Webb 2:30 – 3:00 pmBreak 3:00 – 4:00 pm  Cost driver impact survey results from Oct ‘07  Discussion & wrap-up Gan Wang, Garry Roedler 4:00 – 5:00 pm  Joint meeting w/ SoS cost estimation group Jo Ann Lane

4. Context setting

5. How is Systems Engineering Defined? Acquisition and Supply –Supply Process –Acquisition Process Technical Management –Planning Process –Assessment Process –Control Process System Design –Requirements Definition Process –Solution Definition Process Product Realization –Implementation Process –Transition to Use Process Technical Evaluation –Systems Analysis Process –Requirements Validation Process –System Verification Process –End Products Validation Process EIA/ANSI 632, Processes for Engineering a System, 1999. Note: The requirements of EIA/ANSI 632 are addressed in ISO/IEC 15288, which was also used as a Source for consistent definition in COSYSMO.

6. COSYSMO Origins COSYSMO Systems Engineering (SE) 1950 Software Cost Modeling 1980 CMMI ® 1990 *CMM and CMMI are registered trademarks of Carnegie Mellon University Warfield, J. N., Systems Engineering, United States Department of Commerce PB111801, 1956. Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981. Humphrey, W. Managing the Software Process. Addison-Wesley, 1989. EIA/ANSI 632, Processes for Engineering a System, 1999 ISO/IEC 15288, System Life Cycle Processes, 2002. (Humphrey 1989) (Boehm 1981) (Warfield 1956, EIA 1999, ISO/IEC 2002) SW-CMM ® SE-CMM ® SECM 2000 Current SE Standards EIA-632 ISO/IEC 15288

7. COSYSMO Data Sources BoeingIntegrated Defense Systems (Seal Beach, CA) RaytheonIntelligence & Information Systems (Garland, TX) Northrop GrummanMission Systems (Redondo Beach, CA) Lockheed MartinTransportation & Security Solutions (Rockville, MD) Integrated Systems & Solutions (Valley Forge, PA) Systems Integration (Owego, NY) Aeronautics (Marietta, GA) Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY) General DynamicsMaritime Digital Systems/AIS (Pittsfield, MA) Surveillance & Reconnaissance Systems/AIS (Bloomington, MN) BAE Systems National Security Solutions/ISS (San Diego, CA) Information & Electronic Warfare Systems (Nashua, NH) SAIC Army Transformation (Orlando, FL) Integrated Data Solutions & Analysis (McLean, VA) L-3 Communications Greenville, TX

8. Modeling Methodology 3 rounds; > 60 experts 62 data points; 8 organizations

9. COSYSMO Scope Addresses first four phases of the system lifecycle (adapted from ISO/IEC 15288) Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632) Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

10. COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms + 3 Adj. Factors - Application factors -8 factors - Team factors -6 factors COSYSMO Operational Concept

11. COSYSMO Model Form Where: PM NS = effort in Person Months (Nominal Schedule) A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN} w x = weight for “easy”, “nominal”, or “difficult” size driver = quantity of “k” size driver E = represents diseconomies of scale EM = effort multiplier for the j th cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort.

12. Size Drivers vs. Effort Multipliers Size Drivers: Additive, Incremental –Impact of adding a new item inversely proportional to current size 10 -> 11 rqts = 10% increase 100 -> 101 rqts = 1% increase Effort Multipliers: Multiplicative, system-wide –Impact of adding a new item independent of current size 10 rqts + high security = 40% increase 100 rqts + high security = 40% increase

13. EasyNominalDifficult # of System Requirements0.51.005.0 # of Interfaces1.74.39.8 # of Critical Algorithms3.46.518.2 # of Operational Scenarios9.822.847.4 Size Driver Weights

14. UNDERSTANDING FACTORS –Requirements understanding –Architecture understanding –Stakeholder team cohesion –Personnel experience/continuity COMPLEXITY FACTORS –Level of service requirements –Technology Risk –# of Recursive Levels in the Design –Documentation Match to Life Cycle Needs OPERATIONS FACTORS –# and Diversity of Installations/Platforms –Migration complexity PEOPLE FACTORS –Personnel/team capability –Process capability ENVIRONMENT FACTORS –Multisite coordination –Tool support Cost Driver Clusters Criteria + Matched driver polarity + Grouped by theme + Combined moderately correlated parameters

15. Cost Driver Rating Scales Very LowLowNominalHighVery High Extra HighEMR Requirements Understanding1.871.371.000.770.60 3.12 Architecture Understanding1.641.281.000.810.65 2.52 Level of Service Requirements0.620.791.001.361.85 2.98 Migration Complexity 1.001.251.551.93 Technology Risk0.670.821.001.321.75 2.61 Documentation0.780.881.001.131.28 1.64 # and diversity of installations/platforms 1.001.231.521.87 # of recursive levels in the design0.760.871.001.211.47 1.93 Stakeholder team cohesion1.501.221.000.810.65 2.31 Personnel/team capability1.501.221.000.810.65 2.31 Personnel experience/continuity1.481.221.000.820.67 2.21 Process capability1.471.211.000.880.770.682.16 Multisite coordination1.391.181.000.900.800.721.93 Tool support1.391.181.000.850.72 1.93 EMR = Effort Multiplier Ratio

16. Cost Drivers Ordered by Effort Multiplier Ratio (EMR)

17. Life Cycle Phases/Stages Conceptualize Develop Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle EIA/ANSI 632 Acquisition & Supply Technical Management System Design Product Realization Technical Evaluation Operational Test & Evaluation Effort Profiling

18. Academic prototype Commercial Implementations Proprietary Implementations COSYSMO-R SECOST SEEMaP Impact Academic Curricula Intelligence Community Sheppard Mullin, LLC Policy & Contracts Model 10 theses

19. COSYSMO 2.0 Improvements

20. Recommended Improvements (from user community) 1.Reuse 2.Integration of SwE & SysE estimation 3.Assumption of linearity in COSYSMO cost drivers 4.Effect of cost drivers and scale factors 5.Number of recursive levels of design 6.Risk modeling 7.Establishing best practice guidance 8.Consideration of SoS scope in COSYSMO 9.Estimation in Operation & Maintenance Phase 10.Requirements volatility Joint meeting Deferred

21. 1. Reuse Central question: What is the effect of reuse in estimating systems engineering size/effort? Hypothesis: A COSYSMO reuse submodel will improve the model’s estimation accuracy POC: Jared Fortune References –Valerdi, R., Wang, G., Roedler, G., Rieff, J., Fortune, J., “COSYSMO Reuse Extension,” 22 nd International Forum on COCOMO and Systems/Software Cost Modeling, 2007.

22. 2. Integration of SwE & SysE estimation Central question: What is the overlap between COCOMO II and COSYSMO? Hypothesis: By identifying the WBS elements in COSYSMO that overlap with the WBS in COCOMO II, the systems engineering resource estimation accuracy increases POC: Ricardo Valerdi References –Valerdi, R., The Architect and the Builder: Overlaps Between Software and Systems Engineering. (working paper)

23. 3. Linearity in COSYSMO cost drivers Central question: How do we characterize the non- linearity of cost drivers across the system life cycle? Hypothesis: Not all cost drivers have a constant impact on systems engineering effort throughout the life cycle. POC: Gan Wang References –Wang, G., Valerdi, R., Boehm, B., Shernoff, A., “Proposed Modification to COSYSMO Estimating Relationship,” 18th INCOSE Symposium, June 2008.

24. 4. Effect of cost drivers and scale factors Central question: Can some of the cost drivers become scale factors in the cost estimating relationship calibrated by the new data set? Hypothesis: The current set of size and cost drivers are too sensitive to small variations in rating levels. POC: Gan Wang References –Wang, G., Valerdi, R., Boehm, B., Shernoff, A., “Proposed Modification to COSYSMO Estimating Relationship,” 18th INCOSE Symposium, June 2008. Note from GJR: Potential relationship between this and improvement #7. Also, in reality, number of recursive levels has a different impact on the effort than other cost drivers. This may be able to be addressed by this improvement, by improvement #5, or could be a separate improvement depnding on scope of each. Should be an item for discussion as we go through these.

25. 5. Number of recursive levels of design Central question: How can the integration complexity of system elements one layer below the system-of-interest be operationalized? Hypothesis: The integration complexity of system elements is a predictor of systems engineering effort. POC: John Rieff References –Marksteiner, B., “Recursive Levels and COSYSMO”, October 2007. (working paper)

26. 6. Risk Modeling Central question: How can risk associated with the COSYSMO estimate be quantified? Hypothesis: The output generated by COSYSMO can be quantified using probability distributions for better assessment of the likelihood of meeting the estimate POC: John Gaffney (developer of COSYSMO-R) References –Valerdi, R., Gaffney, J., “Reducing Risk and Uncertainty in COSYSMO Size and Cost Drivers: Some Techniques for Enhancing Accuracy,” 5th Conference on Systems Engineering Research, March 2007, Hoboken, NJ.

27. 7. Best practice guidance for use of Cost Drivers Central question: How can misuse of the COSYSMO cost drivers be avoided? Hypothesis: By developing a best practice guide that describes common pitfalls associated with COSYSMO cost drivers, over-estimation can be reduced or avoided POC: Garry Roedler References –COSYSMO User Manual

28. 8. Consideration of SoS scope in COSYSMO Central question: How can COSYSMO be updated to address system of systems effort estimation? Hypothesis: To be discussed in joint session POC: Jo Ann Lane

29. 9. Estimation in Operation & Maintenance Phase Central question: How can we estimate systems engineering effort in the Operate & Maintain phase? Hypothesis: Coverage of the Operate & Maintenance phases will broaden to model’s life cycle coverage POC: Ricardo Valerdi

30. 10. Requirements volatility Central question: How do we quantify the effects of requirements volatility on systems engineering effort throughout the life cycle? Hypothesis: Requirements volatility is a significant factor for predicting systems engineering effort and can serve as a leading indicator for project success POC: Ricardo Valerdi Feb 15, 2007 Workshop led by Rick Selby –Identified critical success factors in: technical, product, process, people –http://sunset.usc.edu/events/2007/ARR/presentations/RequirementsVol atilityWorkshopSummaryARR2007.ppthttp://sunset.usc.edu/events/2007/ARR/presentations/RequirementsVol atilityWorkshopSummaryARR2007.ppt –Loconsole, A., Borstler, J., “An industrial case study on requirements volatility measures,” 12th Asia-Pacific Software Engineering Conference, 2005.

31. Prioritization Exercise Factors to Consider –Availability of data –Impact on total cost of ownership –Frequency of use –Compatibility with other models (i.e., COCOMO family, PRICE-H, etc.) –Addressal of future trends (Volatility, Uncertainty, Scalability) –Factor interactions

32. Recommended Improvements (from user community) 1.Reuse (completed and approved for V2.0 baseline – see minutes from workshop at PSM User Conference) 2.Integration of SwE & SysE estimation 3.Assumption of linearity in COSYSMO cost drivers 4.Effect of cost drivers and scale factors 5.Number of recursive levels of design 6.Risk modeling (completed and approved for V2.0 baseline – see minutes from workshop at PSM User Conference) 7.Establishing best practice guidance 8.Consideration of SoS scope in COSYSMO 9.Estimation in Operation & Maintenance Phase 10.Requirements volatility Joint meeting Deferred

33. Priority Improve- ment 12345678910 Availability of Data MLLLHL Impact on TOC HHHHMM Frequency of Use HHHHHL Compatible with Models HLLMLM Address Trends LLLMMM Factor Interactions LHHLHL PriorityHM+MMMH L+LL