1. University of Southern California Center for Systems and Software Engineering A Tractable Approach to Handling Software Productivity Domains Thomas Tan Brad Clark 24 th International Forum on COCOMO and Systems/Software Cost Modeling November 3, 2009

2. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling2 Table of Contents Overview Operational Context Software Application Difficulties Productivity Group Matrix –Example Mapping of Systems –Aerospace Project Data Analysis COCOMO II Implications Next Steps Q&A This work is sponsored by the Air Force Cost Analysis Agency

3. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling3 Overview Conventional labels used in Software Productivity Domains use system engineering terms Systems can be comprised of many different software applications, e.g. Command and Control Productivities are influenced by the difficulty of the application type and the operating environment Many applications in a system can exhibit different productivities

4. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling4 Motivation Our motivation is to create a new productivity classification system that can be –Understood by both cost estimators and engineers –Flexible enough to accommodate a wide range of software applications This presentation discusses this new approach As a work-in-progress, we welcome comments and suggestions Please join us at the workshop to discuss this approach and its usage

5. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling5 Overview This approach employs general descriptions to characterize the intended “operational context” (environment) and an application’s “degree of difficulty”: –Operational Context describes the intended environment, platform or target host on which the software application will operate –The Application Difficulty is the degree of difficulty in specifying, developing, and testing a software application

6. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling6 Operational Context Dimensions of constraints include electrical power, computing capacity, storage capacity, repair capability, platform volatility, physical environment accessibility, etc.

7. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling7 Operational Context Very Unconstrained –Unlimited electrical power with backup battery –Regular upgrades and maintenance –Heavily COTS-hardware and COTS-software dependent –Supports multiple users –Limited loss during system down time Unconstrained –Operating environment may move over land and water –Electrical power is limited by mobility (size dependent) –Computing and storage capability depends on ability to carry weight and size –Upgrades and maintenance are not frequent –Components are modular and cannot be disassembled

8. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling8 Operational Context Constrained –Similar to unconstrained in terms of upgrade, maintenance, and component structure –Operating environment may move through the air –Electrical power, computing and storage capability are limited due to weight constraints Very Constrained –Battery is the primary source of electrical power. –Components are miniaturized and custom-made with special tools to access –No ability to upgrade and maintain system after deployment Highly Constrained –Hardware is required to survive in adverse conditions –Limited software control –Real-time sensor inputs and commands –Upgrades and maintenance is not feasible after deployment –Unique system platform that is evolving in maturity as system is being developed

9. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling9 Software Application Difficulties Difficulty would be described in terms of required software reliability, database size, product complexity, integration complexity, information assurance, real- time requirements, different levels of developmental risks, etc.

10. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling10 Software Application Difficulties Very Easy –Risks are well understood with little loss from failure –Business or operational logic is straightforward –Limited interface to other software applications –Mostly stand-alone functionality –Simple tests Easy –Not a new type of application –Risks are understood and mitigation strategies exists –Business or operational logic is straightforward –Requires low reliability due to small or little loss when unavailable –Limited external interface and security requirements Nominal –Somewhat complicated business logic –Risks exists and may need additional study to find mitigation –May require distributed environment with additional security requirements –Moderate, easily recoverable loss for nominal reliability –Not a new type of application

11. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling11 Software Application Difficulties Challenging –High reliability due to greater impact of loss or high probability of risk –Risks are challenging to resolve –Very complicated business logic, external storage may be necessary due to distributed environment –New type of application –Hard real-time control and security requirements –Additional communication interfaces necessary for external components or systems Very Challenging –Extremely complicated business logic –Risks are very challenging to resolve and loss is great (disastrous consequences) –Many automated controls with limited human control –New type of application –Hard real-time control and security requirements –Communication to external components through different interfaces

12. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling12 Productivity Group Matrix Operational Context Application Difficulty Very Uncon- strained Uncon- strainedConstrained Very Constrained Extremely Constrained Very Easy1,11,21,31,41,5 Easy2,12,22,32,42,5 Nominal3,13,23,33,43,5 Challenging4,14,24,34,44,5 Very Challenging 5,15,25,35,45,5 The intersection of an Operational Context rating and Application Difficulty rating represent a Productivity Group –Each group includes the number of project data points, data range, simple cost estimating relationship (of the form: Y = aX b ), standard error, percent bias, and the coefficient of determination, R 2

13. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling13 Example Mapping to Systems Very Unconstrained and Very Easy may include systems like: –Business Analysis Systems –Data Warehousing Systems –Diagnostic Systems Very Unconstrained and Easy may include systems like: –Logistics Systems –Training Systems –Support Systems –Test Systems Constrained and Challenging may include systems like: –Flight Systems –Radar Systems –Signal Processing We welcome comments and suggestions on mappings of systems to the productivity groups.

14. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling14 Data Analysis Experiment Extracted data from Aerospace Corp. report* –188 data records –4 Operating Environments –8 Application Domains Our experiment –Select a Challenging Application: Signal Processing –Select an Easy Application: Test –Calculate estimating relationship (in Y = aX b form), standard error, and R 2 –Our hypothesis is that Easy Applications will have higher productivity than Challenging Applications –Operational Context not considered in this experiment * Gayek, Long, Bell and Larson, “Software Cost and Productivity Model,” Aerospace Report ATR- 2004(8311)-1, The Aerospace Corporation, El Segundo, CA, Feb 04

15. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling15 Aerospace Project Data Analysis Signal Processing –40 records –Estimating relationship for effort Effort = 11.82 x (Size) 0.93 R 2 = 0.7 Standard Error = 0.57

16. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling16 Aerospace Project Data Analysis Test: –16 records –Estimating relationship for effort Effort = 7.46 x (Size) 0.97 R 2 = 0.68 Standard Error = 0.53

17. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling17 Aerospace Project Data Analysis The estimating relationships were used to compare Easy and Challenging Application Difficulty –Size in KSLOC –Effort in PM –Productivity in SLOC/PM Very Easy: TestChallenging: Signal Processing SizeEffortProductivityEffortProductivity 107014410199 20136147192104 50332151449111 100650154856117 20012731571631123 50030961623825131

18. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling18 Challenging Difficulty Plot Very Unconstrained UnconstrainedVery Constrained

19. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling19 Easy Difficulty Plot Very Unconstrained

20. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling20 Productivity Group Matrix – 2 nd Use Productivity Group Matrix is “Cost Model” independent The matrix could be used to “pre-set” parameters for different models, e.g. –COCOMO II –SEER –True-S –SLIM Possibly, the matrix could be extended to have different distributions for effort and duration for lifecycle activities, i.e. –Each cell has its own distribution Operational Context Application Difficulty Very Uncon- strained Uncon- strainedConstrained Very Constrained Extremely Constrained Very Easy1,11,21,31,41,5 Easy2,12,22,32,42,5 Nominal3,13,23,33,43,5 Challenging4,14,24,34,44,5 Very Challenging 5,15,25,35,45,5

21. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling21 COCOMO II Implications Productivity Groups may be used to to pre-set COCOMO II Parameters. This is also a work-in-progress. COCOMO II Parameters Very UnconstrainedUnconstrainedConstrained Very Constrained Extremely Constrained PREDHNLVL RESLHNLVL CPLX-DeviceVLN, LHVHXH PVOLLNHVH TIMENHVHXH STORNHVHXH Pre-sets driven by Operational Context* * These ratings have not been verified with data, they for illustration only

22. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling22 COCOMO II Implications Pre-sets driven by Application Difficulty* FactorsVery EasyEasyNominalChallenging Very Challenging PREDHNLVL RESLHNLVL RELYVLLNHVH CPLX-ControlVLN, LHVHXH CPLX- Computations VLN, LHVHXH CPLX-DataVLN, LHVHXH CPLX-UIVLN, LHVHXH DATALNHVH * These ratings have not been verified with data, they for illustration only

23. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling23 COCOMO II Implications Factors captured by local calibration of COCOMO constants A and B: –Security and Information assurance (given complexity is at least high) Very Easy would have no security or information assurance requirements Very Challenging would require the software to authenticate access, handle denial of service attacks, ensure data integrity, etc. –Components or Sub-system Interactions Heterogeneous components require special interfaces to communicate with each other Homogeneous components are generally built with similar communication interface

24. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling24 COCOMO II Implications COCOMO Parameters not addressed: –Development Flexibility –Team Cohesion –Process Maturity –Developed for Reuse –Analyst Capability –Programmer Capability –Personnel Continuity –Applications Experience –Platform Experience –Language and Tool Experience –Use of Software Tools –Multi-Site Development These are usually driven by business decisions May be indirectly driven by Difficulty or Context

25. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling25 Next Steps The proposed approach to classifying software application productivities by Operational Context and Application Difficulties needs to be verified with the government agencies and industrial communities that will use the approach for software project cost estimates. Solicit input through workshops and document reviews –Verify ratings for operational context and application difficulties. –Assistance in characterization of applications and determining where they fit in the productivity group matrix Analyze project cost data for productivity: each project will be characterized by Operational Context and Application Difficulties If you would like to participate — please give us your contact information

26. University of Southern California Center for Systems and Software Engineering 24th International Forum on COCOMO and Systems/Software Cost Modeling26 Questions? For more information, contact: Thomas Tan thomast@usc.edu@usc.edu 626-617-1128