1. University of Southern California Center for Systems and Software Engineering Next-Generation Software Sizing and Costing Metrics Workshop Report Wilson Rosa, Barry Boehm, Ray Madachy, Brad Clark USC CSSE Annual Research Review March 9, 2010

2. University of Southern California Center for Systems and Software Engineering USC CSSE Annual Research Review - Mar 20102 Topics AFCAA Study Overview Current and Future Challenges for Software Cost Estimation and Data Collection Proposed Metrics Definition Highlights Productivity Data Analysis and Issues for Discussion This work is sponsored by the Air Force Cost Analysis Agency

3. University of Southern California Center for Systems and Software Engineering Project Background Goal is to improve the quality and consistency of estimating methods across cost agencies and program offices through guidance, standardization, and knowledge sharing. Project led by the Air Force Cost Analysis Agency (AFCAA) working with service cost agencies, and assisted by University of Southern California and Naval Postgraduate School We will publish the AFCAA Software Cost Estimation Metrics Manual to help analysts and decision makers develop accurate, easy and quick software cost estimates for avionics, space, ground, and shipboard platforms. 3USC CSSE Annual Research Review - Mar 2010

4. University of Southern California Center for Systems and Software Engineering AFCAA Software Cost Estimation Metrics Manual Chapter 1: Software Estimation Principles Chapter 2: Product Sizing Chapter 3: Product Growth Chapter 4: Effective SLOC Chapter 5: Historical Productivity Chapter 6: Model Calibration Chapter 7: Calibrated SLIM-ESTIMATE Chapter 8: Cost Risk and Uncertainty Metrics Chapter 9: Data Normalization Chapter 10: Software Resource Data Report Chapter 11: Software Maintenance Chapter 12: Lessons Learned Chapter 1: Software Estimation Principles Chapter 2: Product Sizing Chapter 3: Product Growth Chapter 4: Effective SLOC Chapter 5: Historical Productivity Chapter 6: Model Calibration Chapter 7: Calibrated SLIM-ESTIMATE Chapter 8: Cost Risk and Uncertainty Metrics Chapter 9: Data Normalization Chapter 10: Software Resource Data Report Chapter 11: Software Maintenance Chapter 12: Lessons Learned 4USC CSSE Annual Research Review - Mar 2010

5. University of Southern California Center for Systems and Software Engineering Stakeholder Communities Research is collaborative across heterogeneous stakeholder communities who have helped us in refining our data definition framework, domain taxonomy and providing us project data. –Government agencies –Tool Vendors –Industry –Academia SLIM-Estimate ™ TruePlanning ® by PRICE Systems 5USC CSSE Annual Research Review - Mar 2010

6. University of Southern California Center for Systems and Software Engineering Current and Future DoD Cost Estimation Challenges Emergent requirements –Cannot prespecify requirements, cost, schedule, EVMS –Need to estimate and track early concurrent engineering Rapid change –Long acquisition cycles breed obsolescence –DoD Inst 5000.02 emphasis on evolutionary acquisition Net-centric systems of systems –Incomplete visibility and control of elements Model, COTS, service-based, Brownfield systems –New phenomenology, counting rules Always-on, never-fail systems –Need to balance agility and high assurance 6USC CSSE Annual Research Review - Mar 2010

7. University of Southern California Center for Systems and Software Engineering Rapid Change Creates a Late Cone of Uncertainty – Need evolutionary/incremental vs. one-shot development Uncertainties in competition, technology, organizations, mission priorities 7USC CSSE Annual Research Review - Mar 2010

8. University of Southern California Center for Systems and Software Engineering Incremental Development Productivity Decline (IDPD) Example: Site Defense BMD Software –5 builds, 7 years, $100M; operational and support software –Build 1 productivity over 300 LOC/person month –Build 5 productivity under 150 LOC/PM Including Build 1-4 breakage, integration, rework 318% change in requirements across all builds IDPD factor = 20% productivity decrease per build –Similar trends in later unprecedented systems –Not unique to DoD: key source of Windows Vista delays Maintenance of full non-COTS SLOC, not ESLOC –Build 1: 200 KSLOC new; 200K reused@20% = 240K ESLOC –Build 2: 400 KSLOC of Build 1 software to maintain, integrate USC CSSE Annual Research Review - Mar 20108

9. 9 SRDR Data Source

10. University of Southern California Center for Systems and Software Engineering Proposed Metrics Definition Highlights Data quality and standardization issues –No reporting of equivalent “new” code size inputs: Design Modified, Code Modified, Integration Testing Modified, Software Understandability, Programmer Unfamiliarity –No common SLOC reporting – logical, physical, etc. –No standard definitions – Application Domain, Build, Increment, Spiral,… –No common effort reporting – analysis, design, code, test, CM, QA,… –No reporting of quality measures – defect density, defect containment, etc. 10USC CSSE Annual Research Review - Mar 2010

11. University of Southern California Center for Systems and Software Engineering Proposed Metrics Definition Highlights Limited empirical research within DoD on other contributors to productivity besides effort and size: –Operating Environment, Application Domain, and Product Complexity –Personnel Capability –Required Reliability –Quality – Defect Density, Defect Containment –Integrating code from previous deliveries – Builds, Spirals, Increments, etc. –Converting to Equivalent SLOC Reported code sizes for Modified, Unmodified/Reused add no value to a cost estimate unless they translate into equivalent “new” SLOC This research and the resulting Cost Metrics Manual will discuss and address these issues 11USC CSSE Annual Research Review - Mar 2010

12. University of Southern California Center for Systems and Software Engineering USC CSSE Annual Research Review - Mar 201012 SRDR Data Notes: SRDR: Software Resources Data Report Missing Domains: Internet, Maintenance and Diagnostics, Spacecraft bus

13. University of Southern California Center for Systems and Software Engineering Data Analysis Issues USC CSSE Annual Research Review - Mar 201013 Preliminary Results - Do Not Use! PM=15*(EKSLOC) 0.62

14. University of Southern California Center for Systems and Software Engineering Sizing Issues -1 USC CSSE Annual Research Review - Mar 201014 NCSS to Logical SLOC Conversion Ada: 45% C/C++: 61% C#: 61% Java: 72%

15. University of Southern California Center for Systems and Software Engineering Sizing Issues -2 No Modified Code parameters –Percent Design Modified (DM) –Percent Code Modified (CM) –Percent Integration and Test Modified (IM) –Software Understanding (SU) –Programmer Unfamiliarity (UNFM) Program interviews provided parameters for some records USC CSSE Annual Research Review - Mar 201015

16. University of Southern California Center for Systems and Software Engineering Effort Issues Missing effort reporting for different lifecycle phases –Software requirements analysis (REQ) –Software architectural design (ARCH) –Software coding and testing (CODE) –Software integration (INT) –Software qualification testing (QT) –Software management, CM, QA, etc. (Other – very inconsistent) USC CSSE Annual Research Review - Mar 201016

17. Application DomainsVery EasyEasyNominalChallengingVery Challenging Business Systems Large biz system Trillion $/day transaction Internet Simple web pages Web application (shopping) Mega-web application Tools and Tool Systems Verification toolsSafety critical Scientific Systems Offline data reduction Large dataset Simulation and Modeling Low fidelity simulator Physical phenomenon Test and Evaluation Usual Distributed debugging TrainingSet of screens Simulation network Command and Control Taxi-cab dispatch SOS (C4ISR) Mission Management Usual Multi-level security and safety Weapon Delivery and Control Weapon spaceSafety Communications Noise, anomalies handling Radio Safety/Security Frequency- hopping Application Domain Issues 17USC CSSE Annual Research Review - Mar 2010

18. University of Southern California Center for Systems and Software Engineering Proposed SRDR Changes -1 Current SRDRProposed Modifications Application Types (3.7.1 – 17)Reorganize around Operating Environments and Application Domains Add Mission Criticality (add reliability and complexity in a single rating scale) Revisit detailed definitions of the Application Domains Software and External Interface Requirements Add anticipated requirements volatility to 2630- 1, 2 Use percentage of requirements change as volatility input (SRR baseline) Personnel Experience & Turnover Add to 2630-1 Expand years of experience rating scale to 12 years Amount of New (>25%) Modified (<25% mod) Code Add DM, CM, IM, SU, & UNFM factors for modified code Incorporate DM-CM-IM questionnaire Add IM for Reused code Definitions for code types Count at the level it will be maintained USC CSSE Annual Research Review - Mar 201018

19. University of Southern California Center for Systems and Software Engineering Proposed SRDR Changes -2 Current SRDRProposed Modifications Deleted CodeReport deleted code counts Project- or CSCI-level dataSpecify the level of data reporting All Other Direct Software Engineering Development Effort (4.7): Project Management IV&V Configuration Management Quality Control Problem Resolution Library Management Process Improvement Measurement Training Documentation Data Conversion Customer-run Acceptance Test Software Delivery, Installation & Deployment Break into: Management functions Configuration / Environment functions Assessment functions Organization functions (e.g. user & maintainer documentation, measurement, training, process improvement, etc.) USC CSSE Annual Research Review - Mar 201019

20. University of Southern California Center for Systems and Software Engineering Concluding Remarks Goal is to publish a manual to help analysts develop quick software estimates using empirical metrics from recent programs Additional information is crucial for improving data quality across DoD We want your input on Productivity Domains and Data Definitions Looking for collaborators Looking for peer-reviewers Need more data 20USC CSSE Annual Research Review - Mar 2010