Proposed Metrics Definition Highlights Raymond Madachy Naval Postgraduate School CSSE Annual Research Review March 8, 2010

2 Agenda Data Analysis Issues Software Sizing Definitions Recent Workshop Results Conclusions

3 DoD Empirical Data Data quality and standardization issues –No reporting of Equivalent Size Inputs – CM, DM, IM, SU, AA, UNFM, Type –No common SLOC reporting – logical, physical, etc. –No standard definitions – Application Domain, Build, Increment, Spiral,… –No common effort reporting – analysis, design, code, test, CM, QA,… –No common code counting tool –Product size only reported in lines of code –No reporting of quality measures – defect density, defect containment, etc. Limited empirical research within DoD on other contributors to productivity besides effort and size: –Operating Environment, Application Domain, and Product Complexity –Personnel Capability –Required Reliability –Quality – Defect Density, Defect Containment –Integrating code from previous deliveries – Builds, Spirals, Increments, etc. –Converting to Equivalent SLOC Categories like Modified, Reused, Adopted, Managed, and Used add no value unless they translate into single or unique narrow ranges of DM, CM, and IM parameter values. We have seen no empirical evidence that they do…

4 SRDR Data Source

5 Data Collection and Analysis Approach –Be sensitive to the application domain –Embrace the full life cycle and Incremental Commitment Model Be able to collect data by phase, project and/or build or increment Items to collect –SLOC reporting – logical, physical, NCSS, etc. –Requirements Volatility and Reuse Modified or Adopted using DM, CM, IM; SU, UNFM as appropriate –Definitions for Application Types, Development Phase, Lifecycle Model,… –Effort reporting – phase and activity –Quality measures – defects, MTBF, etc.

6 Data Normalization Strategy Interview program offices and developers to obtain additional information not captured in SRDRs… –Modification Type – auto generated, re-hosted, translated, modified –Source – in-house, third party, Prior Build, Prior Spiral, etc. –Degree-of-Modification – %DM, %CM, %IM; SU, UNFM as appropriate –Requirements Volatility -- % of ESLOC reworked or deleted due to requirements volatility –Method – Model Driven Architecture, Object-Oriented, Traditional –Cost Model Parameters – True S, SEER, COCOMO, SLIM

7 Agenda Data Analysis Issues Software Sizing Definitions Recent Workshop Results Conclusions

8 Size Issues and Definitions An accurate size estimate is the most important input to parametric cost models. Desire consistent size definitions and measurements across different models and programming languages The sizing chapters address these : –Common size measures defined and interpreted for all the models –Guidelines for estimating software size –Guidelines to convert size inputs between models so projects can be represented in in a consistent manner Using Source Lines of Code (SLOC) as common measure –Logical source statements consisting of data declarations executables –Rules for considering statement type, how produced, origin, build, etc. –Providing automated code counting tools adhering to definition –Providing conversion guidelines for physical statements Addressing other size units such as requirements, use cases, etc.

9 Sizing Framework Elements Core software size type definitions –Standardized data collection definitions Measurements will be invariant across cost models and data collections venues –Project data normalized to these definitions Translation tables for non-compliant data sources SLOC definition and inclusion rules Equivalent SLOC parameters Cost model Rosetta Stone size translations Other size unit conversions (e.g. function points, use cases, requirements)

10 Core Software Size Types

11 Equivalent SLOC – A User Perspective * “Equivalent” – A way of accounting for relative work done to generate software relative to the code-counted size of the delivered software “Source” lines of code: The number of logical statements prepared by the developer and used to generate the executing code –Usual Third Generation Language (C, Java): count logical 3GL statements –For Model-driven, Very High Level Language, or Macro-based development: count statements that generate customary 3GL code –For maintenance above the 3GL level: count the generator statements –For maintenance at the 3GL level: count the generated 3GL statements Two primary effects: Volatility and Reuse –Volatility: % of ESLOC reworked or deleted due to requirements volatility –Reuse: either with modification (modified) or without modification (adopted) * Stutzke, Richard D, Estimating Software-Intensive Systems, Upper Saddle River, N.J.: Addison Wesley, 2005

12 Adapted Software Parameters For adapted software, apply the parameters: –DM: % of design modified –CM: % of code modified –IM: % of integration required compared to integrating new code –Normal Reuse Adjustment Factor RAF = 0.4*DM + 0.3*CM + 0.3*IM Reused software has DM = CM = 0. Modified software has CM > 0. Since data indicates that the RAF factor tends to underestimate modification effort due to added software understanding effects, two other factors are used: –Software Understandability (SU): How understandable is the software to be modified? –Unfamiliarity (UNFM): How unfamiliar with the software to be modified is the person modifying it?

13 SLOC Inclusion Rules

14 Equivalent SLOC Rules SourceIncludesExcludes New Reused Modified Generated Generator statements 3GL generated statements Converted COTS Volatility How Produced in Development or Source IncludesExcludes New Reused Modified Generated Generator statements (if 3GL generated statements not modified in development) (if 3GL generated statements modified in development) 3GL generated statements (if modified in development) (if not modified in development) Converted COTS Volatility Equivalent SLOC Rules for Development Equivalent SLOC Rules for Maintenance

15 Cost Model Size Inputs

16 Agenda Data Analysis Issues Software Sizing Definitions Recent Workshop Results Conclusions

17 Software Size Type Results Discussions forced clarification of categories and crisper definitions Practical sizing guidance captured in adaptation parameter ranges –E.g. maximum values where adapted code is instead replaced with new software identify range tops Created model-agnostic AAF weight ranges Added sub-categories for generated, converted and translated code to distinguish what is handled for applying equivalent size –Generator statements vs. generated –Translated as-is vs. optimized –Converted as-is vs. optimized 17

18 Software Size Type Results (cont.) Category additions affected SLOC inclusion rules Practical guidance and updated adaption parameter ranges included in AFCAA Software Cost Estimation Metrics Manual Change request for CodeCount to flag and count moved code 18

19 Modified Code Exercise Results 19 * If DM or C M is greater than 50%, start over with new ** IM could be driven by safety critical applications, environments with high reliability requirements

20 Agenda Data Analysis Issues Software Sizing Definitions Recent Workshop Results Conclusions

21 Next Steps Create worked-out exercises for different cases exhibited in sizing rules Incorporate data analysis on existing data to find empirical value ranges for the reuse parameters for each size type in application domains. 21

22 Concluding Remarks Goal is to publish a manual to help analysts develop quick software estimates using empirical metrics from recent programs Additional information is crucial for improving data quality across DoD We want your input on Productivity Domains and Data Definitions Looking for collaborators Looking for peer-reviewers Need more data

23 References United States Department of Defense (DoD), “Instruction , Operation of the Defense Acquisition System”, December W. Rosa, B. Clark, R. Madachy, D. Reifer, and B. Boehm, “Software Cost Metrics Manual”, Proceedings of the 42 nd Department of Defense Cost Analysis Symposium, February B. Boehm, “Future Challenges for Systems and Software Cost Estimation”, Proceedings of the 13 th Annual Practical Software and Systems Measurement Users’ Group Conference, June B. Boehm, C. Abts, W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Upper Saddle River, NJ: Prentice-Hall, R. Stutzke, Estimating Software-Intensive Systems, Upper Saddle River, NJ: Addison Wesley, Madachy R, Boehm B, “Comparative Analysis of COCOMO II, SEER-SEM and True-S Software Cost Models”, USC-CSSE , University of Southern California Center for Systems and Software Engineering, 2008.