CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 1 Los Angeles, CA October 23, 2003 Ricardo Valerdi – USC Center for Software Engineering Working Group Meeting

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 2 COSYSMO Workshop Agenda Morning (8:30 AM – noon) Brief Introduction on COSYSMO Updates on action items Updates on related work Results from Delphi Round 2 Afternoon (1:00 PM – 5:00 PM) SE Sizing discussion Data collection form & NDAs Synchronization with INCOSE activities Open issues Action items

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 3 COSYSMO Introduction Parametric model to estimate system engineering costs Includes 4 size & 14 cost drivers Covers full system engineering lifecycle Developed with USC-CSE Corporate Affiliate and INCOSE participation Conceptualize Develop Oper Test & Eval Transition to Operation Operate, Maintain, or Enhance Replace or Dismantle

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 4 COSYSMO Size Drivers Effort Multipliers Effort Calibration # Requirements # Interfaces # Scenarios # Algorithms + Volatility Factor - Application factors -8 factors - Team factors -6 factors - Schedule driver WBS guided by ISO/IEC COSYSMO Operational Concept

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 5 COCOMO II Software Development phases 20+ years old 200+ calibration points 23 Drivers Variable granularity 3 anchor points Size is driven by SLOC COSYSMO Systems Engineering Entire Life Cycle 2 years old ~3 calibration points 18 drivers Fixed granularity No anchor points Size is driven by requirements, I/F, etc Model Differences

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 6 Size Drivers vs. Effort Multipliers Size Drivers: Additive, Incremental -Impact of adding a new item inversely proportional to current size rqts = 10% increase rqts = 1% increase Effort Multipliers: Multiplicative, system-wide -Impact of adding a new item independent of current size 10 rqts + high security = 40% increase 100 rqts + high security = 40% increase

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 7 4 Size Drivers 1. Number of System Requirements* 2. Number of Major Interfaces 3. Number of Operational Scenarios 4. Number of Critical Algorithms *Weighted by complexity, volatility, and degree of reuse

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 8 Number of System Requirements This driver represents the number of requirements for the system-of-interest at a specific level of design. Requirements may be functional, performance, feature, or service-oriented in nature depending on the methodology used for specification. They may also be defined by the customer or contractor. System requirements can typically be quantified by counting the number of applicable “shall’s” or “will’s” in the system or marketing specification. Do not include a requirements expansion ratio – only provide a count for the requirements of the system-of-interest as defined by the system or marketing specification. EasyNominalDifficult - Well specified- Loosely specified- Poorly specified - Traceable to source- Can be traced to source with some effort - Hard to trace to source - Little requirements overlap - Some overlap- High degree of requirements overlap From 2001

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 9 Number of Major Interfaces This driver represents the number of shared major physical and logical boundaries between system components or functions (internal interfaces) and those external to the system (external interfaces). These interfaces typically can be quantified by counting the number of external and internal system interfaces among ISO/ISE defined system elements. EasyNominalDifficult - Well defined- Loosely defined- Ill defined - Uncoupled- Loosely coupled- Highly coupled - Strong consensus- Moderate consensus- Low consensus - Well behaved- Predictable behavior- Poorly behaved From 2001

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 10 Number of Operational Scenarios This driver represents the number of operational scenarios that a system must satisfy. Such threads typically result in end-to-end test scenarios that are developed to validate the system and satisfy all of its requirements. The number of scenarios can typically be quantified by counting the number of unique end-to-end tests used to validate the system functionality and performance or by counting the number of use case sequence diagrams developed as part of the operational architecture. EasyNominalDifficult - Well defined- Loosely defined- Ill defined - Loosely coupled- Moderately coupled- Tightly coupled or many dependencies/conflicting requirements - Timelines not an issue- Timelines a constraint- Tight timelines through scenario network From 2001

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 11 Number of Critical Algorithms This driver represents the number of newly defined or significantly altered functions that require unique mathematical algorithms to be derived in order to achieve the system performance requirements. As an example, this could include a complex aircraft tracking algorithm like a Kalman Filter being derived using existing experience as the basis for the all aspect search function. Another example could be a brand new discrimination algorithm being derived to identify friend or foe function in space-based applications. The number can be quantified by counting the number of unique algorithms needed to support each of the mathematical functions specified in the system specification or mode description document. EasyNominalDifficult - Existing algorithms- Some new algorithms- Many new algorithms - Basic math- Algebraic by nature- Difficult math (calculus) - Straightforward structure- Nested structure with decision logic - Recursive in structure with distributed control - Simple data- Relational data- Persistent data - Timing not an issue- Timing a constraint- Dynamic, with timing issues - Library-based solution- Some modeling involved- Simulation and modeling involved From 2001

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling Cost Drivers 1. Requirements understanding 2. Architecture understanding 3. Level of service requirements 4. Migration complexity 5. Technology Maturity 6. Documentation Match to Life Cycle Needs 7. # and Diversity of Installations/Platforms 8. # of Recursive Levels in the Design Application Factors (8)

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 13 Requirements understanding This cost driver rates the level of understanding of the system requirements by all stakeholders including the systems, software, hardware, customers, team members, users, etc. Very lowLowNominalHighVery High Poor, unprecedented system Minimal, many undefined areas Reasonable, some undefined areas Strong, few undefined areas Full understanding of requirements, familiar system EMR = 1.73

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 14 Architecture understanding This cost driver rates the relative difficulty of determining and managing the system architecture in terms of platforms, standards, components (COTS/GOTS/NDI/new), connectors (protocols), and constraints. This includes tasks like systems analysis, tradeoff analysis, modeling, simulation, case studies, etc. Very lowLowNominalHighVery High Full understanding of architecture, familiar system and COTS Strong understanding of architecture and COTS, few undefined areas Reasonable understanding of architecture and COTS, some weak areas Minimal understanding of architecture and COTS, many undefined areas Poor understanding of architecture and COTS, unprecedented system 2 level WBS3-4 level WBS5-6 level WBS>6 level WBS EMR = 1.66

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 15 Level of service requirements This cost driver rates the difficulty and criticality of satisfying the ensemble of level of service requirements, such as security, safety, response time, interoperability, maintainability, the “ilities”, etc. Viewpoint Very lowLowNominalHighVery High DifficultySimpleLow difficulty, coupling Moderately complex, coupled Difficult, coupled KPPs Very complex, tightly coupled CriticalitySlight inconvenience Easily recoverable losses Some lossHigh financial loss Risk to human life EMR = 2.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 16 Migration complexity This cost driver rates the complexity of migrating the system from previous system components, databases, workflows, environments, etc., due to new technology introductions, planned upgrades, increased performance, business process reengineering, etc. Very lowLowNominalHighVery High Introduction of requirements is transparent Difficult to upgradeVery difficult to upgrade EMR = 1.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 17 Technology Maturity The maturity, readiness, and obsolescence of the technology being implemented. Immature or obsolescent technology will require more System Engineering effort. ViewpointVery LowLowNominalHighVery High MaturityStill in the laboratory Ready for pilot use Proven on pilot projects and ready to roll-out for production jobs Proven through actual use and ready for widespread adoption Technology proven and widely used throughout industry ReadinessConcept defined (TRL 3 & 4) Proof of concept validated (TRL 5 & 6) Concept has been demonstrated (TRL 7) Concept qualified (TRL 8) Mission proven (TRL 9) Obsolescen ce - Technology is outdated and use should be avoided in new systems - Spare parts supply is scarce - Technology is stale - New and better technology is on the horizon in the near-term - Technology is the state-of-the- practice - Emerging technology could compete in future EMR = 2.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 18 Documentation match to life cycle needs The breadth and depth of documentation required to be formally delivered based on the life cycle needs of the system. ViewpointVery lowLowNominalHighVery High BreadthGeneral goals Broad guidance, flexibility is allowed Streamlined processes, some relaxation Partially streamlined process, some conformity with occasional relaxation Rigorous, follows strict customer requirements DepthMinimal or no specified documentati on and review requirements relative to life cycle needs Relaxed documentation and review requirements relative to life cycle needs Amount of documentation and reviews in sync and consistent with life cycle needs of the system High amounts of documentation, more rigorous relative to life cycle needs, some revisions required Extensive documentation and review requirements relative to life cycle needs, multiple revisions required EMR = 1.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 19 # and diversity of installations/platforms The number of different platforms that the system will be hosted and installed on. The complexity in the operating environment (space, sea, land, fixed, mobile, portable, information assurance/security). For example, in a wireless network it could be the number of unique installation sites and the number of and types of fixed clients, mobile clients, and servers. Number of platforms being implemented should be added to the number being phased out (dual count). ViewpointNominalHighVery High Sites/installationsSmall # of installations or many similar installations Moderate # of installations or some amount of multiple types of installations Large # of installations with many unique aspects Operating environment Not a driving factorModerate environmental constraints Multiple complexities/constraints caused by operating environment PlatformsFew types of platforms (< 5) being installed and/or being phased out/replaced Modest # and types of platforms (5 < P <10) being installed and/or being phased out/replaced Many types of platforms (> 10) being installed and/or being phased out/replaced Homogeneous platformsCompatible platformsHeterogeneous, incompatible platforms Typically networked using a single protocol Typically networked using several consistent protocols Typically networked using different protocols EMR = 1.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 20 # of recursive levels in the design The number of levels of design related to the system-of-interest (as defined by ISO/IEC 15288) and the amount of required SE effort for each level. ViewpointVery LowLowNominalHighVery High Number of levels >7 Required SE effort Ad-hoc effortMaintaining system baseline with few planned upgrades Sustaining SE for the product line, introducing some enhancements of product design features or optimizing performance and/or cost Maintaining multiple configurations or enhancements with extensive pre-planned product improvements or new requirements, evolving Maintaining many configurations or enhancements with extensive pre- planned product improvements, new requirements rapidly evolving EMR = 1.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling Cost Drivers (cont.) 1. Stakeholder team cohesion 2. Personnel/team capability 3. Personnel experience/continuity 4. Process capability 5. Multisite coordination 6. Tool support Team Factors (6)

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 22 Stakeholder team cohesion Represents a multi-attribute parameter which includes leadership, shared vision, diversity of stakeholders, approval cycles, group dynamics, IPT framework, team dynamics, trust, and amount of change in responsibilities. It further represents the heterogeneity in stakeholder community of the end users, customers, implementers, and development team. ViewpointVery LowLowNominalHighVery High Culture  Stakeholders with diverse expertise, task nature, language, culture, infrastructure  Highly heterogeneous stakeholder communities  Heterogeneous stakeholder community  Some similarities in language and culture  Shared project culture  Strong team cohesion and project culture  Multiple similarities in language and expertise  Virtually homogeneous stakeholder communities  Institutionalized project culture Communication  Diverse organizational objectives  Converging organizational objectives  Common shared organizational objectives  Clear roles & responsibilities  High stakeholder trust level EMR = 1.50

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 23 Personnel/team capability Basic intellectual capability of a Systems Engineer to analyze complex problems and synthesize solutions. Very LowLowNominalHighVery High 15 th percentile35 th percentile55 th percentile75 th percentile90 th percentile Personnel experience/continuity The applicability and consistency of the staff at the initial stage of the project with respect to the domain, customer, user, technology, tools, etc. Very lowLowNominalHighVery High ExperienceLess than 2 months1 year continuous experience, other technical experience in similar job 3 years of continuous experience 5 years of continuous experience 10 years of continuous experience Annual Turnover 48%24%12%6%3% EMR = 2.15 EMR = 2.0

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 24 Process capability Maturity per CMMI or EIA 731. Very lowLowNominalHighVery HighExtra High CMMILevel 1 (lower half) Level 1 (upper half) Level 2Level 3Level 4Level 5 EIA731Performed SE process, activities driven only by immediate contractual or customer requirements, SE focus limited Managed SE process, activities driven by customer and stakeholder needs in a suitable manner, SE focus is requirements through design Defined SE process, activities driven by benefit to program, SE focus is through operation Quantitativel y Managed SE process, activities driven by SE benefit, SE focus on all phases of the life cycle Optimizing SE process, continuous improvement, activities driven by system engineering and organizational benefit, SE focus is product life cycle & strategic applications EMR = 1.60

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 25 Multisite coordination Location of stakeholders, team members, resources, corporate collaboration barriers. ViewpointVery lowLowNominalHighVery HighExtra High Collocation International, severe time zone impact Multi-city and multi- national, considerable time zone impact Multi-city or multi- company, some time zone effects Same city or metro area Same building or complex, some co- located stakeholders or onsite representation Fully co- located stakeholder s Communications Some phone, mail Individual phone, FAX Narrowband Wideband electronic communicatio n Wideband electronic communication, occasional video conference Interactive multimedia Corporate collaboration barriers Severe export and security restrictions Mild export and security restrictions Some contractual & Intellectual property constraints Some collaborative tools & processes in place to facilitate or overcome, mitigate barriers Widely used and accepted collaborative tools & processes in place to facilitate or overcome, mitigate barriers Virtual team environmen t fully supported by interactive, collaborativ e tools environmen t EMR = 1.68

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 26 Tool support Coverage, integration, and maturity of the tools in the Systems Engineering environment. Very lowLowNominalHighVery High No SE toolsSimple SE tools, little integration Basic SE tools moderately integrated throughout the systems engineering process Strong, mature SE tools, moderately integrated with other disciplines Strong, mature proactive use of SE tools integrated with process, model-based SE and management systems EMR = 1.87

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 27 Action Items from Keystone (7/03) 1.Develop “short form” version of the Delphi instrument 2.Update application domain categories to include INCOSE and DoD 3.Include more commercially-oriented examples in Delphi form 4.Update Size drivers to reflect data collection effort vs. cost estimation effort 5.Incorporate new direction of IEEE 1220, ISO/IEC 15288, and EIA/ANSI 632 life cycle stages 6.Follow up with BAE’s SE estimation efforts in New Hampshire 7.Continue work on “Technology Maturity” driver and include NASA/DARPA Technology Readiness Levels 8.Update “# of Critical Algorithms” driver

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 28 Updates on Related Work 1.COSYSMO for Trade Studies 2.myCOSYSMO tool version SE Sizing overview paper

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 29 USC/Raytheon myCOSYSMO* *Developed by Gary Thomas at Raytheon Garland

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 30 Delphi Round 2 Participants (36) Paul MohlmanAerospace Abe Santiago*Aerospace Anh TuAerospace Maurie HartmanBoeing Rob FloweDAU Denton Tarbet*Galorath Gary HafenLMCO Rick EdisonLMCO Garry Roedler*LMCO Carl NewmanLMCO Bob BeckleyLMCO Trish PerssonLMCO George WaltherLMCO Steven WongNorthrop Grumman Steve CopoloffNorthrop Grumman Joe KiernickiNorthrop Grumman Ed TseNorthrop Grumman Bill SlobodianNorthrop Grumman Lori VaughanNorthrop Grumman Gary Thomas*Raytheon Randy CaseRaytheon Bob VojtechRaytheon John Rieff*Raytheon Greg CahillRaytheon Larry S KlecknerRaytheon Deke Dunlap*Raytheon Ron TownsendRaytheon Stan ParsonRaytheon Tony Jordano*SAIC Charley ZumbaSAIC Jim ArmstrongSPC Sarah SheardSPC Chris MillerSPC Barry Boehm*USC Don Reifer*USC George FriedmanUSC *also participated in Round 1

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 31 Delphi Survey Results Suggested Rel. EffortDelphi Respondents EMR Easy – Nominal – Difficult Rel. EffortStandard Deviation # Requirements 0.5 – 1.0 – – 1.5 – – 1.7 – 13.1 # Interfaces 1.0 – 3.0 – – 3.8 – – 2.0 – 3.5 # of Ops Scen 14.0 – 29.0 – – 21.8 – – 13.8 – 30.5 # Algorithms 3.0 – 6.0 – – 5.3 – – 2.4 – 6.7 Ratings for Size Drivers

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 32 Delphi Round 2 Highlights (cont.) Range of sensitivity for nominal Size Drivers # Algorithms # Requirements # Interfaces # Scenarios 5.3 Relative Effort Rel. Effort 0.6 – 1.5 – – 3.8 – – 21.8 – – 5.3 – # Requirements # Interfaces # of Ops Scen # Algorithms

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 33 Delphi Survey Results Suggested EMRDelphi Respondents EMR EMRStandard Deviation Reqs Und Arch Und Level of Serv Req Migration Complx Tech Maturity Documentation # Install/Platforms # levels in design Ratings for Cost Drivers (Application Factors)

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 34 Delphi Survey Results Suggested Rel. EffortDelphi Respondents EMR EMRStandard Deviation Stakeholder Coh Pers/team Cap Pers exp/cont Process cap Multisite Coord Tool Support Ratings for Cost Drivers (Team Factors)

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 35 Delphi Round 2 Highlights (cont.) Range of sensitivity for Cost Drivers (Application Factors) EMR Requirements und. # of levels in design Level of service reqs.Technology MaturityArch. understanding # of Install/platforms Documentation Migration complexity 1.84

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 36 Delphi Round 2 Highlights (cont.) Range of sensitivity for Cost Drivers (Team Factors) EMR Pers experience/cont Stakeholder cohesion Pers/team capability Tool support Multisite coordination Process capability

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 37

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 38

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 39

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 40

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 41 Systems Engineering Sizing # Requirements System Level (too high): The system shall provide notification of out-of- tolerance inputs and outputs to authorized parties. File Level (about right): Each system component has one or more files of parameters to monitor, report exceptions, and adjust tolerances in a familiar way. Parameter Level (too low): The system shall determine that the temperature T at point P, is between T11 and T12, and report exceptions to the safety monitor. # Interfaces For hardware interfaces, use Number and Complexity of External Interface files. For user interfaces, use Number and Complexity of Input Files, Output Files, External Queries.

CII Forum – 10/22/03 18th International Forum on COCOMO and Software Cost Modeling 42 Questions or Comments? Dr. Barry Boehm Ricardo Valerdi Website