I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force Next-Generation Systems and Software Cost Estimation Wilson Rosa Technical Advisor Air Force Cost Analysis Agency (AFCAA) October 28, 2008

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Problem Statement  Emerging technologies such as Systems of Systems (SoS) Model Driven Architecture (MDA) Enterprise Resource Planning (ERP) Service-Oriented Architecture (SOA) Commercial Off the Shelf (COTS) Services Reuse-Driven (RUSE) are complicating AFCAA's job of producing accurate software cost estimates 2

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 3 Next-Generation System Challenges 1. Lines of Code not appropriate for  Model Driven Architecture  COTS-Based Systems (SOA, ERP, etc.) 2. No guidelines for estimating beyond software design  Infrastructure (servers, LAN, routers, etc.)  Concurrent Users  Enterprise Services (collaboration, discovery, portal, etc.)  Data Migration, External Interfaces  Interoperability and Interdependency 3. Unfamiliar with total system cost drivers 4. Lack of Empirical Research – SOA, ERP, SoS, MDA

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Data Challenges  AFCAA has multiple software datasets  Unable to combine software datasets because of inconsistencies and lack of standardization  Schedule seems to be reported at program and not CSCI level -- all CSCI’s have same schedule  No reporting of % re-design, % re-coding, % re-test  No common counting method – logical, physical, etc.  No standard application type definitions  No common code counting tool  Product size only reported in lines of code  No reporting of COCOMO, SEER, PRICE parameters  No reporting of quality measures – defects, MTBF, etc. 4

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 5 Parametric Model Challenges  Most AF program offices rely on cost models  Have these models been calibrated with the most recent DoD data? ( )  Most models only cover a fraction, not total system costs – infrastructure, users, etc.  Model calibration with recent data will help reduce the program office estimating error rate

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Consequence: Significant Cost Growth (%) 6 StatisticsTotal System*Software Only** Minimum-64%-80% Mean45%37% Median27%8% High471%623% Standard Deviation71%107% Milestone PhaseDevelopment Sample Size Year of Data Source : *John McCrillis, 36 th DOD Cost Analysis Symposium (2003) **Defense Automated Cost Information System

I n t e g r i t y - S e r v i c e - E x c e l l e n c e Headquarters U.S. Air Force 7 Software Cost Metrics Manual OVERVIEW

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 8 Scope  Cost Agencies in conjunction with University of Southern California will publish a manual to help analysts develop quick software estimates using reliable metrics from recent programs

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 9 Software Cost Manual Content Chapter 1: Basic Software Estimation Chapter 2: Product Size Estimation Chapter 3: Schedule and Cost Growth Chapter 4: Default ESLOC Parameters Chapter 5: Historical Productivity Chapter 6: Default COCOMO/SEER/True-S Parameters Chapter 7: SLIM-ESTIMATE Calibration Chapter 8: Risk and Uncertainty Parameters Chapter 9: Data Normalization Chapter 10: Space Software Cost Guide Chapter 11: Software Maintenance

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 10 Data Sources CommoditySourceFormatYearProjectsCSCIs Space, Ground, AirSoftware Resource Data Reports DD-Form SpaceAEHF SEER AirF-22 EMD and Increment II Boeing SpaceMILSTAR SEER1990s428 SpaceFAB-T DD-Form SpaceNPOESS SEER SpaceTSAT DD-Form Air, GroundNorthrop Grumman COCOMO, SEER Space, GroundRaytheon COCOMO Air, Ship, GroundNaval Center for Cost Analysis TECHNOMICS AirLockheed Martin COCOMO AirArmy Cost and Economics Analysis Center TECHNOMICS GroundFuture Combat System DD-Form Space, IntelIntelligence Community (NRO, NGA, DNI) SEERTBD Space, GroundAerospace, Space & Missile System Center Unknown TBD SpaceNASA JPL Unknown TBD Space, Air, GroundUSC AffiliatesUnknown TBD >168>598 Note: Expecting over 1600 CSCIs by 2010

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 11 Data Normalization  USC will interview program offices and developers to obtain additional information… 1. Cost Model Parameters – True S, SEER, COCOMO 2. Reuse Type – auto generated, re-hosted, translated, modified 3. Reuse Source – in-house, third party 4. Degree-of-Modification – %DM, %CM, %IM 5. Method – Model Driven Architecture, Object-Oriented, Traditional  Available Data 1. DoDAF – System Views, Operational Views, etc. 2. Software Resource Data Report – Software Size, Effort, Schedule 3. Cost Analysis Requirements Description (CARD)  System Description, Users, Infrastructure, locations, interfaces, etc.

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Chapter 4: ESLOC Parameters 12 Reuse TypeReuse Source Design Modified Code Modified Integration ModifiedESLOC Auto GeneratedIn-House 0% 50%15% Third Party 0% 100%30% Re-HostIn-House 0% 100%30% Third Party 0%24%100%37% TranslatedIn-House 0%100% 60% Third Party 15%100% 66% ModifiedIn-House 0%100% 60% Third Party 100% UnmodifiedIn-House 0% 32%10% Third Party 0% 100%30%  Default values from recent programs  Based on Reuse Type and Reuse Source

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 13 Chapter 5: Historical Productivity  Overview and Guidelines  Historical Productivity Dataset by Application  Default Productivity Ranges by Application IOC CSCIApplication Productivity (ESLOC/MM) Raw (KSLOC) ESLOC Effort (MM) Peak Effort (FTE) Schedule (Months) 1999Signal Processing Avionics Spot Antenna Control Payload BootstrapBus

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Significance of Proposed Manual  Collected data can be used for  Systems of Systems cost research  Cost Model improvement initiatives  SEER, True-S, COCOMO, etc.  Understanding relationships between Next- Generation Processes and COCOMO cost drivers can encourage researchers to explore new strategies to improve available cost models… 14

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED Way Ahead  Short Term  NCCA/AFCAA Software Cost Handbook (Nov 2008)  Software Cost Metrics Manual (2009)  Long Term  Cost Risk and Uncertainty Handbook Volume II (2011)  ERP Cost Guide (2010)  Impact of MDA on Cost Modeling (2010)  SOA Cost Study (2012)  Data Center Cost Study (2010) 15 Note: Any data you provide will not be attributed to your company or program, but will be combined with like data from other sources and generic zed"

I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 16 Backup Slides I n t e g r i t y - S e r v i c e - E x c e l l e n c e