1. University of Southern California Center for Systems and Software Engineering Building Cost Estimating Relationships for Acquisition Decision Support Brad Clark, Ray Madachy, Thomas Tan, & Barry Boehm Wilson Rosa, Sponsor

2. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling22 Topics Research problem and objectives Data challenges and resolution Results Future work Project led by the Air Force Cost Analysis Agency (AFCAA) working with service cost agencies, and assisted by University of Southern California and Naval Postgraduate School November 3, 2010

3. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling3 Problem For many years, there have been efforts to collect data from multiple projects and organizations –Data Analysis Center for Software (DACS) –Software Engineering Information Repository (SEIR) –International Software Benchmarking Standards Group (ISBSG) –Large Aerospace Mergers (Attempts to create company-wide databases) –USAF Mosemann Initiative (Lloyd Mosemann Asst. Sec. USAF) –USC CSSE COCOMO II repository –DoD Software Resources Data Report (SRDR) Purpose: to derive estimating relationships and benchmarks for size, cost, productivity and quality All have faced common challenges such as data definitions, completeness and integrity November 3, 2010

4. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling4 Data Analysis Cost = a * X b Research Objectives Using SRDR data, improve the quality and consistency of estimating methods across cost agencies and program offices through guidance, standardization, and knowledge sharing. –Characterize different Application Domains and Operating Environments within DoD –Analyze collected data for simple Cost Estimating Relationships (CER) within each domain –Develop rules-of-thumb for missing data Make collected data useful to oversight and management entities Data RecordsCERs November 3, 2010

5. University of Southern California Center for Systems and Software Engineering SRDR Raw Data (520 observations) 25th International Forum on COCOMO and Systems/Software Cost Modeling5 PM = 1.67 * KSLOC 0.66 November 3, 2010

6. University of Southern California Center for Systems and Software Engineering Data Conditioning Segregate data Normalize sizing data (predictor) Normalize effort data (response) Address multi-build data 25th International Forum on COCOMO and Systems/Software Cost Modeling6November 3, 2010

7. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling77 SRDR Data Segmentation November 3, 2010

8. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling8 Communication Domain Analysis-1 EnvironmentExamplesBrief Definition Fixed Ground Computing facilities Command and Control centers Tactical Information centers Communication centers Manned and unmanned fixed, stationary land sites (buildings) with access to external power sources, backup power sources, physical access to systems, regular upgrades and maintenance to hardware and software, support for multiple users. Possible noisy environment. DomainExamplesBrief Definition Communications Radios Microwave controller Large telephone switching systems Network management Software that controls the transmission and receipt of voice, data, digital and video information. The software operates in real-time or in pseudo real-time. Environment: Fixed ground, mobile ground, manned and unmanned airborne, or unmanned space. November 3, 2010

9. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling9 Mission Management Analysis-1 EnvironmentExamplesBrief Definition Avionics Fixed-wing aircraft Helicopters Manned airborne platforms. Software that is complex and runs in real-time in embedded computer systems. It must often operates under interrupt control to process timelines in the nanoseconds. DomainExamplesBrief Definition Mission Management Operational Flight Program Mission Computer Flight Control Software Software that enables and assists the operator in performing mission management activities including scheduling activities based on vehicle, operational and environmental priorities. Environment: Mobile ground, avionics or manned space. November 3, 2010

10. University of Southern California Center for Systems and Software Engineering Normalizing Size Normalize the SLOC counting method to Logical SLOC –Physical SLOC count converted to Logical SLOC count by programming language –Non-comment SLOC count converted to Logical SLOC count by programming language Convert Auto-Generated SLOC convert to Equivalent SLOC (ESLOC) –Use AAF formula: (DM% * 0.4) + (CM% * 0.3) + (IM% * 0.3) –DM = CM = 0; IM = 100 Convert Reused SLOC to ESLOC with AAF formula –DM = CM = 0; IM = 100 Convert Modified SLOC to ESLOC –Use AAF formula: (DM% * 0.4) + (CM% * 0.3) + (IM% * 0.3 –Default values: Low – Mean – High based on 90% confidence interval Create Equivalent SLOC count and scale to thousands (K) to derive EKSLOC (New + Auto-Gen+ Reused+ Modifed) / 1000 = EKSLOC Remove all records with an EKSLOC below 1.0 25th International Forum on COCOMO and Systems/Software Cost Modeling10November 3, 2010

11. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling11 SLOC Count Conversion Experiment 11 Logical SLOC = 0.611 * NCSS Count R 2 = 0.9974 Logical SLOC = 0.611 * NCSS Count R 2 = 0.9974 November 3, 2010

12. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling12 SLOC Count Conversion Factors Data Count Total Line to Logical NCSS to Logical Ada40.250.52 C/C++120.320.61 C#80.350.68 Java60.350.72 Perl40.530.70 PHP40.440.66 Overall380.330.64 12 For example, (C++ NCSS SLOC Count) * 0.61 = (C++ Logical SLOC Count) November 3, 2010

13. University of Southern California Center for Systems and Software Engineering Convert Modified Size to ESLOC Use AAF formula: (DM% * 0.4) + (CM% * 0.3) + (IM% * 0.3) Problems with missing DM, CM & IM in SRDR data Program interviews provided parameters for some records For missing data, use records that have data in all fields to derive recommended values for missing data 25th International Forum on COCOMO and Systems/Software Cost Modeling13 November 3, 2010

14. University of Southern California Center for Systems and Software Engineering Convert Modified Size to ESLOC Communication Domain (18 observations) Mission Management (19 observations) 25th International Forum on COCOMO and Systems/Software Cost Modeling14 DM%CM%IM% Median152864 Low 90% CL142049 Mean253162 High 90% CL364275 DM%CM%IM% Median100 Low 90% CL586976 Mean758388 High 90% CL9297100 November 3, 2010

15. University of Southern California Center for Systems and Software Engineering Normalizing Effort Labor hours are reported for 7 categories: –Software Requirements –Software Architecture (including Detailed Design) –Software Code (including Unit Testing) –Software Integration and Test –Software Qualification Test –Software Developmental Test & Evaluation –Other (Mgt, QA, CM, PI, etc.) Create effort distribution percentages for records that have hours in requirements, architecture, code, integration and qualification test phases (developmental test evaluation and other phases may or may not be blank) Fill in missing hours using effort distribution table Currently don’t use Developmental Test and Other hours 25th International Forum on COCOMO and Systems/Software Cost Modeling15November 3, 2010

16. University of Southern California Center for Systems and Software Engineering Distribution Percentages Communication (27 observations) Mission Management (16 observations) 25th International Forum on COCOMO and Systems/Software Cost Modeling16 Req’t%Arch%Code%I&T%QT% Median162732214 Low 90% CL142329174 Mean172732207 High 90% CL2030352310 Req’t%Arch%Code%I&T%QT% Median2413341711 Low 90% CL181127126 Mean2414321713 High 90% CL3019372220 November 3, 2010

17. University of Southern California Center for Systems and Software Engineering Multi-Build Data 25th International Forum on COCOMO and Systems/Software Cost Modeling17November 3, 2010

18. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling18 One More: Team Experience SRDR Data Definition –Report the percentage of project personnel in each category –Highly Experienced in the domain (three or more years of experience) –Nominally Experienced in the project domain (one to three years of experience) –Entry-level Experienced (zero to one year of experience) Need to include Team Experience (TXP) in CERs to estimate cost After analyzing the data, the following quantitative values are assigned: –Highly experienced: 0.60 –Nominally experienced: 1.00 –Entry-level experienced: 1.30 November 3, 2010

19. University of Southern California Center for Systems and Software Engineering Data Conditioning Results 25th International Forum on COCOMO and Systems/Software Cost Modeling19 Just Kidding! November 3, 2010

20. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling20 Five Phases PM = 6.35 * EKSLOC 0.939 R 2 = 0.86 Five Phases PM = 6.35 * EKSLOC 0.939 R 2 = 0.86 November 3, 2010

21. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling21 Three Phases PM = 3.8 * EKSLOC 0.953 R 2 = 0.88 Three Phases PM = 3.8 * EKSLOC 0.953 R 2 = 0.88 November 3, 2010

22. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling22 Five Phases PM = 5.06 * EKSLOC 1.22 R 2 = 0.890 Five Phases PM = 5.06 * EKSLOC 1.22 R 2 = 0.890 November 3, 2010

23. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling23 Three Phases PM = 3.47 * EKSLOC 1.19 R 2 = 0.9094 Three Phases PM = 3.47 * EKSLOC 1.19 R 2 = 0.9094 November 3, 2010

24. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling24 Simple Cost Estimating Relationships Notes: CER: Cost Estimating Relationship PM: Person Months (152 labor hours / month) EKSLOC: Equivalent Thousands of Source Lines of Code R2: Correlation Coefficient that ranges for 0 to 1 Bias: Average percentage error that estimate is above/below actual value CommunicationsMission Management CER PM = 3.8 * EKSLOC 0.95 * TXPPM = 3.47 * EKSLOC 1.19 * TXP # Data Pts 2636 EKSLOC Range 4.8 to 2001.2 to 201 R2R2 0.880.91 November 3, 2010

25. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling25 Conclusion Workshop: Thursday from 8:00 AM to 2:50 PM Come find out –How to use the information to construct an estimate –Next steps and schedule –Conclusions about this approach Discussion on how the SRDR may change –We made recommendations for improvements We will also discuss ranking Application Domains by order of productivity November 3, 2010

26. University of Southern California Center for Systems and Software Engineering 25th International Forum on COCOMO and Systems/Software Cost Modeling26 Questions? For more information, contact: Wilson Rosa Wilson.Rosa@pentagon.af.mil 703-604-0395 Or Brad Clark bkclark@csse.usc.edu 703-754-0115 Or Ray Madachy rjmadach@nps.edu November 3, 2010