1. Estimation, Planning & Control Processes Can Make the Difference Between Project success and Failure Dan Galorath galorath@galorath.com Copyright 2011 Galorath Incorporated

2. Introduction Estimating is critical for all kinds of systems Yet many treat is as a second rate process Having a repeatable estimation process is critical to both estimating AND to successful projects Estimation and measurement go hand in hand While Galorath addresses Hardware, Software, IT, and manufacturing Context of this talk will be largely from software © 2009 Copyright Galorath Incorporated 2

3. Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR) Problem: Humans seem hardwired to be optimists Optimism from cognitive biases & organizational pressures Exaggerate talents & degree of control Attribute negative consequences to external factors Anchoring ( relying too heavily on one piece of information ) magnifies optimism Most pronounced for new initiatives Solution: Temper with outside view Supplement traditional forecasting w/ statistical Parametrics Dont remove optimism, but balance optimism & realism

4. Example of Tempering: (Source How To Measure Anything) German Mark V Tanks Allies estimated production by analyzing serial numbers Used the captured tanks as a random sample and predicted probability of various production levels

5. An Estimate Defined An estimate is the most knowledgeable statement you can make at a particular point in time regarding: Effort / Cost Schedule Staffing Risk Reliability Estimates more precise with progress A WELL FORMED ESTIMATE IS A DISTRIBUTION 5

6. Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5 Ariane 5 video Note: $500 Million payload Failure due to reused software from (Ariane 4) with incompatible assumptions for exception condition that was not required the culture within the Ariane program… only addressed random hardware failures… which can quite rationally be handled by a backup system the view had been taken that software should be considered correct until it is shown to be at fault! Why should we care: Software cost & schedule should be sufficient for successful missions

7. Software Is A Key Risk Item In Weapons Systems Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to drastically underused… GAO GAO identified 42 programs at risk for cost & schedule 1.military requirements changes 2.software development challenges 3.workforce issues National Institute of Standards and Technology (NIST) Software defects cost nearly $60 Billion Annually 80% of development costs involve identifying and correcting defects Software, not Hardware or technology readiness levels were called out

8. Software Is Getting More Complex Source: (Impact of Weapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF) Why should we care: Software is already difficult, costly, and error prone. More complexity means more problems

9. Software Is Providing an Increasing Percentage of Functionality Software development has been problematic for military aircraft developers Consuming about 40 percent of the USAFs RDT&E budget Average software schedule overrun was 222 percent of the original estimate Overruns often justified by stating weapon systems performance requirements are evermore demanding And thus cause greater reliance on software Software provided about 10 percent of an F-4s functionality in the 1960s but provides over 80 percent of an F-22s Why should we care: unrealistic cost estimates lead to unrealistic budgets and un-executable programs Meier Study… DAU

10. Software Failures Are Costly and Dangerous December 23, 2004, internal memo by Deputy Secretary of Defense called for the cancellation of the U.S. Air Forces C-130J transport aircraft to cut its losses on the overpriced, unneeded, and problem-plagued C-130J. Estimated cancellation cost were $1.78 Billion – in 2004 dollars! The loss of a classified satellite after only 7 seconds on orbit prompted the review of software and processors that has caused the most recent delay and a potential $1 billion overrun in Lockheed Martin's Space-Based Infrared System (SBIRS). Strategic Command is worried about potential gaps in coverage for some mission areas in part because satellites are being delivered later than planned. Iraqi Scud missile slipped through Patriot missile defenses a year ago and hit U.S. Army barracks in Dhahran, Saudi Arabia, killing 28 servicemen.

11. Cutter Consortium Software Project Survey: 62% overran original schedule by more than 50% 64% more than 50% over budget; 70% had critical product quality defects after release Standish Group CHAOS Report 46% challenged 19% failed 35% successful ~$875 billion spent on IT ~$300 billion spent on IT projects ~$57 billion wasted annually IT Has Similar Failures

12. Estimate Total Ownership Costs; 60+% Can Be Infrastructure & Services Software Development Software Maintenance IT Infrastructure IT Services © 2009 Copyright Galorath Incorporated 12

13. Poor Estimates Effect on Projects Inaccurate estimates can reduce project success: Poor implementations Critical processes dont scale Emergency staffing Cost overruns caused by underestimating project needs Scope creep Forever changing project goals Frustration Customer dissatisfaction Cost overruns and missed schedules Project Failures Important project business decisions made early with minimum knowledge & maximum uncertainty 13 Why should we care: Poor estimates and plans are a root cause of program failure

14. Manual Estimates: Human Reasons For Error (Metrics Can Help) Manual Task estimates yield SIGNIFICANT error Desire for credibility motivates overestimate behavior (80% probability?) So must spend all the time to be reliable Better approach force 50% probability & have buffer for overruns Technical pride sometimes causes underestimates 14

15. SEER: Software Analysis Tools A Complete Software Project Management Solution Knowledge & Experience Baseline(s) & Final Actuals To-date Actuals Measure & Analyze Monitor & Control Baseline Estimate & Plan A Foundation of Risk Management Quantitative Project Management 15

16. 16 10 Step Software Estimation Process: Consistent Processes = Reliable Estimates 1.Establish Estimate Scope 2.Establish Technical Baseline, Ground Rules, Assumptions 3.Collect Data 4.Estimate and Validate Software Size 5.Prepare Baseline Estimates 7.Quantify Risks and Risk Analysis 6.Review, Verify and Validate Estimate 8.Generate a Project Plan 9.Document Estimate and Lessons Learned 10.Track Project Throughout Development

17. 17 This chapter discusses a 1972 GAO report on cost estimating We reported that cost estimates were understated and causing unexpected cost growth Many of the factors causing this problem are still relevant today We also discuss a 12 step process for producing high quality cost estimates GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them

18. 18 GAO Publication: Why cost estimates are required for government programs and challenges developing credible results Introduces why cost estimates are required for government programs Developing annual budgets, supporting management decisions about which program to fund, and evaluating resource requirements at key decision points Discusses various challenges associated with developing credible results

19. © 2010 Galorath Incorporated Contractors At Least Level 3 Would Be Acceptable Level 0 Informal or no estimating Manual effort estimating without a process Level 1 Direct Task Estimation Spreadsheets Ad Hoc Process Level 2 Formal Sizing (e.g. function points) Direct Task Estimation Simple model (Size * Productivity) or informal SEER Use Some measureme nt & analysis Informal Process Level 3 Formal Sizing Robust Parametric estimation (SEER) Estimate vs. actual capture Formalized Multiple Estimate Process Rigorous measurement & analysis Parametric planning & Control Risk Management Repeatable process Level 4 Formal sizing Repeatable process Robust parametric estimating (SEER) Rigorous measurement & analysis Parametric estimation with tracking & control Risk Management Process improvement via lessons learned Level 5 Formal sizing Repeatable process Robust parametric estimating (SEER) Rigorous measurement & analysis Parametric estimation with tracking & control Risk Management Continuous process improvement Why should we care? Maturity is related to estimate viability… Better estimation process more likely to be successful in execution

20. Balancing Resources & Schedule Is A Science For a given Size, Complexity and Technology Impossible Minimum Time To Complete (Effort Increases to Reduce Schedule ) Work Expands To Fill Time (Effort Increases due to lack of pressure ) Inefficient Minimum TimeOptimal Effort (Lower Effort for Longer Schedule) Calendar Time Effort Months Effort Increase due to Longer Schedule

21. Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs) 21

22. Understanding & Tracking Defects, Growth And Other Metrics Heath and Status Indicator shows status and trends from the previous snapshot Including Size Growth and Defect Discovery/Removal Rate yellowUser defined control limits to control the transition between red-yellow-green Track defect discovery and removal rates against expected rates Increased defect reporting rate shows a worsening trend Track software size growth 22

23. Considering Maintenance During Planning Can Yield More Successful Long Term Results 23

24. Software & IT Systems Are About Business Value CostValue DevelopmentMaintenanceIT InfrastructureIT Services Return on Investment Internal Rate of Return Net Present Value

25. Data Doesnt Have To Be Perfect To Be Useful: But Is Has To Be Viable 80 Calories per serving 2.5 Servings per can 4 Ounces, Condensed, 8 Ounces With Water

26. You have an estimate … Now what? 26

27. Correlation does not imply causation Just because two data points may sit side by side doesnt mean they are the same or will have the same outcome Casual analysis is a recognized error in medicine Perhaps ??? The Error of Causal Analysis Creating a False Association Tumor Can Cause Headache Headache doesnt mean a tumor 27

28. Use Historical Measurement to evaluate your estimate! Its easy to dig deeper and deeper to justify an estimate! 28

29. 29 Normalization Adjustments for homogeneity Time phased data can be particularly difficult to handle As spent dollars are influenced by inflation Requires knowledge of time period involved May require converting to fixed dollar units Process data Need to know specific sub-phase of the process May require allocation to sub-phase

30. 30 Normalization: What To Look For In An Actual Phase : All activities may not be included. System Concept missing Integration missing Phases System ConceptsSystem Req & Design System Req AnalysisPreliminary Design Detailed DesignCode / Unit Testing Software TestSystem Integration / OT&E

31. 31 Normalization: What To Look For In An Actual: Labor Labor : All categories may not be included. Quality Assurance missing Configuration Mgt missing Labor Categories ManagementSystem Engineering DesignCoding Data PreparationTest Configuration Mgmt.Quality Assurance

32. Estimation Best Practices Decide Why You Want An Estimate Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity Have A Documented, Repeatable Estimation Process Make The Estimating Process As Simple As Possible; But No Simpler Be Proactive: The Process Is Important, The Tools Go Along With The Process Get Buy-in From Program Managers Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them Tie The Estimate To The Plan © 2009 Copyright Galorath Incorporated 32

33. Estimation Best Practices 2 Evaluate Total Ownership Cost; Not Just Development Estimate A Range And Pick A Point For The Plan Re-estimate The Program When It Changes Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale Keep A History: Start An Enterprise Database NOW… Business Case: Evaluate ROI In Addition To Costs Convert Expert Spreadsheets Into A Common Language © 2009 Copyright Galorath Incorporated 33

34. Estimation Best Practices 3 Track Progress Vs. Estimate Throughout The Life Cycle Estimate Schedule As Well As Effort (Cost) For Complete Picture Tie The Business Case Into The Estimating Process Attack Non-productive Rework As Part Of The Process © 2009 Copyright Galorath Incorporated 34

35. Estimation Best Practices 4 Have clear definitions: What does complete mean. What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.) Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.) Measure what you care about Estimating & tracking rework can help control costs Dont ignore IT infrastructure and IT services costs Tracking defect sources can go along with the process © 2009 Copyright Galorath Incorporated 35

36. Additional Information www.galorath.com Dan on estimating BLOG: www.galorath.com/wpwww.galorath.com/wp Email: galorath@galorath.comgalorath@galorath.com © 2009 Copyright Galorath Incorporated 36

37. Economics Is a Key Driver In Software (Source: Royce) © 2009 Copyright Galorath Incorporated 37