2. Copyright 2005 Northrop Grumman Corporation Measuring the Benefits of Mature Processes 20th International Forum on COCOMO and Software Cost Modeling 24 October 2005 Rick Hefner, Ph.D. Northrop Grumman Corporation

3. Copyright 2005 Northrop Grumman Corporation 1 Background  Many organizations have implemented process improvement efforts (CMM, CMMI, ISO, Six Sigma) to become better, cheaper and faster  Some organizations have not realized the quantitative or return- on-investment (ROI) benefits reported in the literature Are the literature claims of ROI true? Are their tricks for getting better ROI? What are the timelines for realizing these benefits? CMM® and CMMI® are registered trademarks of Carnegie Mellon University

4. Copyright 2005 Northrop Grumman Corporation 2 Agenda  Principles of process improvement  Industry data on ROI  Issues surrounding the measurement of benefits  Strategic actions needed to achieve maximum ROI  Northrop Grumman lessons learned

5. Copyright 2005 Northrop Grumman Corporation 3 Software Projects Have Historically Suffered from Mistakes Reference: Steve McConnell, Rapid Development People-Related Mistakes 1. Undermined motivation 2. Weak personnel 3. Uncontrolled problem employees 4. Heroics 5. Adding people to a late project 6. Noisy, crowded offices 7. Friction between developers and customers 8. Unrealistic expectations 9. Lack of effective project sponsorship 10. Lack of stakeholder buy-in 11. Lack of user input 12. Politics placed over substance 13. Wishful thinking Process-Related Mistakes 14. Overly optimistic schedules 15. Insufficient Risk Management 16. Contractor failure Insufficient planning 17. Abandonment of planning under pressure 18. Wasted time during the fuzzy front end 19. Shortchanged upstream activities 20. Inadequate design 21. Shortchanged quality assurance 22. Insufficient management controls 23. Premature or too frequent convergence 25. Omitting necessary tasks from estimates 26. Planning to catch up later 27. Code-like-hell programming Product-Related Mistakes 28. Requirements gold-plating 29. Feature creep 30. Developer gold-plating 31. Push me, pull me negotiation 32. Research-oriented development Technology-Related Mistakes 33. Silver-bullet syndrome 34. Overestimated savings from new tools or methods 35. Switching tools in the middle of a project 36. Lack of automated source-code control Standish Group, 2003 survey of 13,000 projects 34% successes 15% failures 51% overruns Standish Group, 2003 survey of 13,000 projects 34% successes 15% failures 51% overruns

6. Copyright 2005 Northrop Grumman Corporation 4 Many Approaches to Solving the Problem  Which weaknesses are causing my problems?  Which strengths may mitigate my problems?  Which improvement investments offer the best return? People Product Technology Tools Management Structure Business Environment Process Methods One solution!

7. Copyright 2005 Northrop Grumman Corporation 5 Approaches to Process Improvement Data-Driven (e.g., Six Sigma, Lean)  Clarify what your customer wants (Voice of Customer)  Critical to Quality (CTQs)  Determine what your processes can do (Voice of Process)  Statistical Process Control  Identify and prioritize improvement opportunities  Causal analysis of data  Determine where your customers/competitors are going (Voice of Business)  Design for Six Sigma Model-Driven (e.g., CMM, CMMI)  Determine the industry best practice  Benchmarking, models  Compare your current practices to the model  Appraisal, education  Identify and prioritize improvement opportunities  Implementation  Institutionalization  Look for ways to optimize the processes

8. Copyright 2005 Northrop Grumman Corporation 6 Typical Data-Driven Results Cited in Literature YearRevenue ($B)Invested ($B)Savings ($1B)% Revenue Motorola 1986-2001356.9(e)ND164.5 Allied Signal 199815.1ND0.53.3 GE 199679.20.2 199790.80.411.1 1998100.50.51.31.2 1999111.60.621.8 1996-1999382.11.64.41.2 Honeywell 199823.6ND0.52.2 199923.7ND0.62.5 200025.0ND0.72.6 1998-200072.3ND1.82.4 Ford 2000-200243.9ND12.3 http://www.isixsigma.com/library/content/c020729a.asp

9. Copyright 2005 Northrop Grumman Corporation 7 Typical Model-Driven Results Cited in Literature 10:1 Defect Reduction 50% Productivity Increase More Accurate Estimates Rework Drops From 54% to 4% “Benefits of CMM-Based Process Improvement”, Herbsleb et al., Software Engineering Institute, 1994 CategoryRange Yearly cost of process improvement activities $49K - $1,202K Years engaged in SPI 1 – 9 Cost of SPI per engineer $490 - $2,004 Productivity gain per year 9% - 67% Early detection gain per year (defects discovered pre-test) 6% - 25% Yearly reduction in time to market 15% - 23% Yearly reduction in post-release defects 10% - 94% Business value (ROI) 4.0 - 8.8

10. Copyright 2005 Northrop Grumman Corporation 8 Typical CMMI Benefits Cited in Literature  Reduced costs  33% decrease in the average cost to fix a defect (Boeing)  20% reduction in unit software costs (Lockheed Martin)  Reduced cost of poor quality from over 45 percent to under 30 percent over a three year period (Siemens)  Faster Schedules  50% reduction in release turnaround time (Boeing)  60% reduction in re-work following test (Boeing)  Increase from 50% to 95% the number of milestones met (General Motors)  Greater Productivity  25-30% increase in productivity within 3 years (Lockheed Martin, Harris, Siemens)  Higher Quality  50% reduction of software defects (Lockheed Martin)  Customer Satisfaction  55% increase in award fees (Lockheed Martin) “Demonstrating the Impact and Benefits of CMMI: An Update and Preliminary Results,” Software Engineering Institute, CMU/SEI-2003-SR-009, Oct 2003

11. Copyright 2005 Northrop Grumman Corporation 9 The Knox Cost of Quality Model  Extension of the Cost of Quality model used in manufacturing CostCategoryDefinitionTypical Costs for Software ConformanceAppraisalDiscovering the condition of the product Testing and associated activities, product quality audits PreventionEfforts to ensure product quality SQA administration, inspections, process improvements, metrics collection and analysis Non- conformance Internal failures Quality failures detected prior to product shipment Defect management, rework, retesting External failures Quality failures detected after product shipment Technical support, complaint investigation, defect notification Knox’s Theoretical Model for Cost of Software Quality (Digital Technical Journal, vol.5, No. 4., Fall 1993, Stephen T. Knox.)

12. Copyright 2005 Northrop Grumman Corporation 10 Knox Model – Theoretical Benefits COCOMO also predicts ~10% increase in productivity for each increase in CMMI Level

13. Copyright 2005 Northrop Grumman Corporation 11 Why Measuring ROI is Hard  What do you count as “Investment”?  Training? QA? Data gathering? New practices?  What would we have done instead?  What do you count as “Savings?  Increased predictability – what’s the value?  Increased productivity – who gets the benefit?  Better competitive position – how measured?  How do you measure the change?  Multiple causes – awareness, knowledge, infrastructure  Short-term vs. long-term – Hawthorne effect  Over what time-frame? See also The Shangri-La of ROI”, Sarah Sheard and Christopher Miller, Software Productivity Consortium ROI = Savings ----------------- Investment

14. Copyright 2005 Northrop Grumman Corporation 12 Process Effectiveness Program Effectiveness Mission Assurance Operations Effectiveness Dashboards for Enterprise- Wide Measurement Communications & Best-Practice Sharing Robust Governance Model (Policies, Processes, Procedures) Risk Management Systems Engineering Independent Reviews Training, Tools, & Templates CMMI Level 5 for Software, Systems, and Services ISO 9001 and AS-9100 Certification Six Sigma Northrop Grumman Mission Systems Approach Mission Success Requires Multiple Approaches

15. Copyright 2005 Northrop Grumman Corporation 13 Process Effectiveness CMMI & Six Sigma courses Policies & processes course Standard Training Modules for each job function: engineering, project management, QA, CM, etc. Staff Competence & Training Communications & Collaboration Process Asset Library Audits & Appraisals Assuring mission success by making the people and processes more informed and effective 5 13 Northrop Grumman sites externally appraised at CMMI Level 5

16. Copyright 2005 Northrop Grumman Corporation 14 Program Effectiveness  Six Sigma connects process improvement and business value  Six Sigma projects can help focus and measure CMMI-driven process improvements  Identify the customer’s needs, maximize the value/cost  Tools for management by variation (CMMI Levels 4 and 5)  Results to date  4000 Green Belts, 200 Black Belts, 12 Master Black Belts  500 completed Six Sigma projects, 250 in progress  Significant benefit to our customer – lower costs, better performance Charter team, map process & specify CTQs Measure process performance Identify & quantify root causes Select, design & implement solution Institutionalize improvement, ongoing control DEFINE MEASURE ANALYZE CONTROL IMPROVE Assuring mission success by identifying the customer’s needs and reducing defects

17. Copyright 2005 Northrop Grumman Corporation 15 Operational Effectiveness Assuring mission success by providing independent cost, schedule and risk realism Tying it all together Product Process Capability Data Repository Software, Hardware, Accounting Productivity, Defects, Maintenance Phase Relationships,Systems Engineering Functions Lessons Learned People Presence Tools Programs Process Risk Analysis Cost Estimates Cost Estimation Relationships Program Benchmarking Life Cycle Productivity Analysis Software Sizing and Modeling Predictive Modeling Quantitative Management Parametric Modeling Expertise DOD Software Industry Expertise Risk and Predictive Modeling Analysis Certified Function Point Specialists Six Sigma Black Belts Professional Society Board Members Active on Government Working groups Key participants on Milestone reviews) Commercial Modeling Tools Northrop Grumman Developed Tools Monitor, Manage, Report, Update and Calibrate Structured Project Reporting, Training Standardization of Data, Metrics Manual, Approval CMMI Measurement,

18. Copyright 2005 Northrop Grumman Corporation 16 Lessons Learned  Model-driven and data-driven process improvements compliment each other  Model-driven improvements are difficult to measure precisely  Long improvement cycles and broad focus make it difficult to isolate cause and effect  Substantial anecdotal evidence of significant ROI  Data-drive improvements are more easily measured  Short improvement cycles, narrow focus  Efforts concentrate data, measurement systems, tie improvements to business goals