Presentation is loading. Please wait.

Presentation is loading. Please wait.

2012 Australian Cost Conference Keynote Cost Estimation: A Key Component of Affordable Program Success Dan Galorath Copyright 2012.

Similar presentations

Presentation on theme: "2012 Australian Cost Conference Keynote Cost Estimation: A Key Component of Affordable Program Success Dan Galorath Copyright 2012."— Presentation transcript:

1 2012 Australian Cost Conference Keynote Cost Estimation: A Key Component of Affordable Program Success Dan Galorath Copyright 2012 Galorath Incorporated

2 Key Points Cost is a key project performance parameter Viable affordability decisions yield project achievements Cost Estimation with repeatable process is best practice 2 © 2012 Copyright Galorath Incorporated 2

3 © 2012 Copyright Galorath Incorporated 3 Using Best Practices Are A Best Practice Themselves Method or technique that has consistently shown results superior to those achieved with other means, and that is used as a benchmark Best" practice can evolve to become better as improvements are discovered Processes Tools People

4 © 2012 Copyright Galorath Incorporated 4 Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR) Problem: Humans seem hardwired to be optimists Optimism from cognitive biases & organizational pressures Exaggerate talents & degree of control Attribute negative consequences to external factors Anchoring ( relying too heavily on one piece of information ) magnifies optimism Most pronounced for new initiatives Best practice: Temper with outside view Supplement traditional forecasting w/ statistical parametrics Dont remove optimism, but balance optimism & realism

5 © 2012 Copyright Galorath Incorporated 5 While Optimism Needs Tempering, So Does Short Sightedness (Source Northrop)

6 Key Points Cost is a key project performance parameter © 2012 Copyright Galorath Incorporated 6

7 © 2012 Copyright Galorath Incorporated 7 Cost Overruns Are Everywhere GAO: Staggering cost overruns dwarf modest improvements in Defense acquisition R&D costs of weapons programs increased 42% over original estimates Average delay 22 months in delivering initial capabilities Evolving technical requirements Shortage of qualified government staff to manage "Every dollar of cost growth on a DoD weapon system represents a lost opportunity to pay for another national priority" The Collins Class submarine program: Murphy was an optimist BG Group shares hit by $5.4B cost overruns on Australian Liquified Natural Gas project

8 © 2012 Copyright Galorath Incorporated 8 Software Is A Key Risk Item In Weapons Systems Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to be drastically underused… GAO GAO identified 42 programs at risk for cost & schedule 1.military requirements changes development challenges 3.workforce issues National Institute of Standards and Technology (NIST) Software defects cost nearly $60 Billion Annually 80% of development costs involve identifying and correcting defects Software, not Hardware or technology readiness levels were called out

9 © 2012 Copyright Galorath Incorporated 9 Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR) Case Study: Levi Strauss $5M ERP deployment contracted Risks seemed small Difficulty interfacing with customers systems Had to shut down production Unable to fill orders for 3 weeks $192.5M charge against earnings on a $5M IT project failure IT projects touch so many aspects of organization they pose a new singular risk

10 © 2012 Copyright Galorath Incorporated 10 An ROI Analysis of A New System: Should We Fund This? Can we do better? Will stakeholders tolerate a loss for 3 years? What is the risk?

11 © 2012 Copyright Galorath Incorporated 11 Software & IT Systems Are About Business Value Total Ownership Cost Value DevelopmentMaintenance IT Infrastructure & Services Technical Debt Return on Investment Internal Rate of Return Net Present Value Software economics should provide methods for analyzing the choices software projects must make. Leon Levy Business economics should provide methods for analyzing choices in which projects to fund

12 © 2012 Copyright Galorath Incorporated 12 Potentially 80% of Projects Dont Return Adequate Value Most projects cost more than they return, Mercer Consulting : When the true costs are added up, as many as 80% of technology projects actually cost more than they return. It is not done intentionally but the costs are always underestimated and the benefits are always overestimated. Dosani, 2001

13 © 2012 Copyright Galorath Incorporated 13 Cutter Consortium Software Project Survey: 62% overran original schedule by more than 50% 64% more than 50% over budget 70% had critical product quality defects after release Standish Group CHAOS Report 46% challenged 19% failed 35% successful Fully one in six of the projects we studied was a black swan, with a cost overrun of 200% on average, and a schedule overrun of almost 70% ~$875 billion spent on IT ~$300 billion spent on IT projects ~$57 billion wasted annually IT Has Similar Failures

14 Using Gates and Refining Estimates is a Best Practice (Adapted from K. Aguanno) Great Idea Great Idea Opportunity Analysis Preliminary Business Case Committed Business Case Achieve Business Case Achieve Business Case Concept - Describe idea & Possible benefits Marketing Analysis Feasibility Study Pilot or Proof of Concept -Determine customer acceptance -Interview focus groups, etc. - Design solution - Estimate cost / schedule - Analyze risk - Determine feasibility / ROI - Validate & commit to design & approach -Revised estimates & schedule -Risk reduction -Baselined plan -Build solution - Deploy -Achieve -business case -Capture lessons learned -Including estimating Full Execution or Deployment Risk Gate 1 Gate 2 Gate 3 Gate 4 Investment in time and money

15 © 2012 Copyright Galorath Incorporated 15 Causes of Project Failure Source: POST Report on UK Government IT Projects Lack of a clear link between the project and the organisations key strategic priorities, including agreed measures of success. 1. Lack of clear senior management and ministerial ownership and leadership2. Lack of effective engagement with Stakeholders3. Lack of skills and proven approach to project management and risk management 4. Lack of understanding of and contact with the supply industry at senior levels within the organisation 5. Evaluation of proposals driven by initial price rather than long-term value for money (especially securing the delivery of business benefits) 6. Too little attention to breaking development and implementation into manageable steps 7. Inadequate resources and skill to deliver the total delivery portfolio8. Source: POST Report on UK Government IT Projects

16 © 2012 Copyright Galorath Incorporated 16 June 28, 2010 Mandate September 14, 2010 Guidance November 3, 2010 Implementation Five Specific Areas of Concern: Target Affordability and Control Cost Growth Reduce Non-Productive Processes and Bureaucracy Incentivize Productivity and Innovation in Industry Promote Real Competition Improve Tradecraft in Services Acquisition US Better Buying Power Initiatives

17 Affordability Initiatives With Should Cost and Will Cost Should Cost Performance Cost Initiatives (Applied practices & improvements) Cost Initiatives (Applied practices & improvements) Will Cost Performance Will Cost Performance - = © 2012 Copyright Galorath Incorporated 17

18 Characteristics of a Successful Should Cost Review (Source: AT Kearney) © 2012 Copyright Galorath Incorporated 18

19 Key Points Cost Estimation with repeatable Process is best practice © 2012 Copyright Galorath Incorporated 19

20 © 2012 Copyright Galorath Incorporated 20 An Estimate Defined An estimate is the most knowledgeable statement you can make at a particular point in time regarding: Effort / Cost Schedule Staffing Risk Reliability Estimates more precise with progress A WELL FORMED ESTIMATE IS A DISTRIBUTION 20

21 © 2012 Copyright Galorath Incorporated 21 Viable Estimation Is Critical Estimating is critical for all kinds of systems Yet many treat is as a second rate process Everyone estimates…. Just most get it wrong and dont have a process Having a repeatable estimation process is critical to both estimating AND to successful projects Estimation and measurement go hand in hand

22 © 2012 Copyright Galorath Incorporated 22 Estimation Methods 1 of 2 Model Category DescriptionAdvantagesLimitations GuessingOff the cuff estimates Quick Can obtain any answer desired No Basis or substantiation No Process Usually Wrong Analogy Compare project with past similar projects. Estimates are based on actual experience. Truly similar projects must exist Expert Judgment Consult with one or more experts. Little or no historical data is needed; good for new or unique projects. Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent. Top Down Estimation A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component. Provides an estimate linked to requirements and allows common libraries to size lower level components. Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation.

23 © 2012 Copyright Galorath Incorporated 23 Estimation Methods 2 of 2 Model CategoryDescriptionAdvantagesLimitations Bottoms Up Estimation Divide the problem into the lowest items. Estimate each item… sum the parts. Complete WBS can be verified. The whole is generally bigger than the sum of the parts. Costs occur in items that are not considered in the WBS. Design To Cost Uses expert judgment to determine how much functionality can be provided for given budget. Easy to get under stakeholder number. Little or no engineering basis. Simple CERs Equation with one or more unknowns that provides cost / schedule estimate. Some basis in data. Simple relationships may not tell the whole story. Historical data may not tell the whole story. Comprehensive Parametric Models Perform overall estimate using design parameters and mathematical algorithms. Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable. Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation. 23

24 Best Value Data Needs (adapted from Boeing Value Front) Models & Methods Customer Need Priorities (Decision Analysis) Customer Desirability Attribute Value O/S Dev Prod TOC Cost-Risk Attribute Value Distribution Uncertainty Simulation © 2012 Copyright Galorath Incorporated 24

25 © 2012 Copyright Galorath Incorporated 25 Functional Focus Example: Ladies Purse Function ………………………………………... Hold stuff Cost…………………………………………… $400 at Nordstrom What else will perform the function? Paper bag - Cost = $0.05 Go to plastic bag for more durability Cost = $0.10 Add color………………………………………..Cost = $0.15 Add strap.……………………………………….Cost = $0.25 Misses one component of customer satisfaction

26 © 2012 Copyright Galorath Incorporated 26 Far Out Higher TRL Level Estimation Goal: Better Cost For Highly Advanced Space Missions (15-20 Years in the Future) Critical items at less than TRL 4… Like asking Edison in 1876 How much longer for the light bulb? Hard to say In 1879, once he had found a workable carbon filament, How much will a production version of the light bulb cost to develop and produce Tom? Then a TRL 4 question Proposed Hyperspectral Imaging Satellite predicted fielding: 2016 This capability would be of interest to: Military space asset planners Government agencies Commercial satellite producers Advanced concept designers

27 © 2012 Copyright Galorath Incorporated 27 Black Swans (Unknowns) The match is unlikely to ever be perfect because some projects are affected by unknown unknowns, also called Black Swans, that is, events that are essentially unpredictable (e.g., a severe worldwide credit squeeze) Score Overrun %

28 © 2012 Copyright Galorath Incorporated 28 Dealing With the Problem of Assumptions Assumptions are essential but… Incorrect assumptions can drive an estimate to uselessness Use an assumption verification process 1. Identify assumptions 2. Rank order assumptions based on estimate impact 3. Identify high ranking assumptions that are risky 4. Clarify high ranking, high risk assumptions & quantify what happens if those assumptions change 5. Adjust range of SEER inputs to describe the uncertainty in assumptions Estimates must have assumptions defined, but… Bad assumptions should not be justification for bad estimates

29 © 2012 Copyright Galorath Incorporated 29 System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK If you cant tell me what it is, I cant tell you what it costs. -Mike Jeffers If you can tell me the range of what it might be, I can tell you the range of cost, schedule & probability. -Dan Galorath

30 © 2012 Copyright Galorath Incorporated 30 Uncertainty in the Cost Depends On Uncertainty of the Project Itself Even though the entire project may be highly uncertain, tasks to the next gate should be estimatible within 10%. SEER includes uncertainty in its estimates Within 10%

31 31 Statistician Drowns in River with Average Depth of 3 Feet! © 2012 Copyright Galorath Incorporated 31

32 Range vs. Point Estimates (Source US Army) Range of Risk & Uncertainty Estimating Accuracy Trumpet Technical and Program Maturity ROM -30% to +75% Parametric -10% to +20% Analogy -15% to +30% Engineering -5% to +15% Actual -3% to +10% Target Cost Point estimate is most likely within range estimate with higher potential for cost increase Range estimate provides a degree of risk and uncertainty Upper bound Lower bound +75% -30% B A Materiel Solution Analysis Systems Acquisition Sustainment Technology Development Production & Deployment Operations & Support Engineering and Manufacturing Development C Pre-Systems Acquisition © 2012 Copyright Galorath Incorporated 32

33 © 2012 Copyright Galorath Incorporated What is likely to happen Feel lucky? Firm Fixed Price? Understand the risk before you commit!

34 © 2012 Copyright Galorath Incorporated 34 Dealing With Early Estimates Give a range: but they will hear the bottom of the range Give a high probability number: Will still be low in some cases and may be high in many cases but consider it a probably not to exceed Sticker shock may be a problem Give a category rather than an number: e.g over $10m Cat 1; over $5m Cat 2; over $1m Cat 3, etc. Stakeholders always remember the first number even when told it is preliminary. Developers will be optimistic by nature unless the process is tempered.

35 © 2012 Copyright Galorath Incorporated 35 Potential Accuracy at Various Gates (Adapted From Canada Treasury) Estimate Accuracy Is a Function of Input Information Quality. Estimates Can Be Much Closer than Shown IF Data Is Available.

36 © 2012 Copyright Galorath Incorporated 36 Converting Uncertainty to Risk Is A Best Practice Many treat as synonymous but… In risk situations, probabilities assigned based on data In uncertainty situations, we may assign probabilities But there is no data to back them up If probability of rain tomorrow is 50%, thats a risk We have historical data & scientific analyses about rain which make it reasonable to estimate the probability If we say Probability of success in harnessing nuclear fusion for routine energy production within the next ten years is 50%, thats just a guess about uncertaintyevidence for the assignment is lacking or very weak If you toss this thumbtack, what is the probability it will land this way instead of on its back?

37 This chapter discusses a 1972 GAO report on cost estimating We reported that cost estimates were understated and causing unexpected cost growth Many of the factors causing this problem are still relevant today We also discuss a 12 step process for producing high quality cost estimates GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them © 2012 Copyright Galorath Incorporated 37

38 Generalized 10 Step System Estimation Process Establish Estimate Scope 2.Establish Technical Baseline, Ground Rules, Assumptions 4.Refine Technical Baseline Into Estimable Components 4.Collect data / estimation inputs 5.Estimate Baseline Cost, Schedule, Affordability Value 6.Validate Business Case Costs & Benefits (go / no go) 6.Quantify Risks and Risk Analysis 8.Generate a Project Plan 9.Document Estimates and Lessons Learned 10.Track Project Throughout Development © 2012 Copyright Galorath Incorporated 38

39 Contractors At Least Level 3 Would Be Acceptable Level 0 Informal or no estimating Manual effort estimating without a process Level 1 Direct Task Estimation Spreadsheets Ad Hoc Process Level 2 Formal Sizing (e.g. function points) Direct Task Estimation Simple model (Size * Productivity) or informal SEER Use Some measureme nt & analysis Informal Process Level 3 Formal Sizing Robust Parametric estimation (SEER) Estimate vs. actual capture Formalized Multiple Estimate Process Rigorous measurement & analysis Parametric planning & Control Risk Management Repeatable process Level 4 Formal sizing Repeatable process Robust parametric estimating (SEER) Rigorous measurement & analysis Parametric estimation with tracking & control Risk Management Process improvement via lessons learned Level 5 Formal sizing Repeatable process Robust parametric estimating (SEER) Rigorous measurement & analysis Parametric estimation with tracking & control Risk Management Continuous process improvement Why should we care? Maturity is related to estimate viability… Better estimation process more likely to be successful in execution © 2012 Copyright Galorath Incorporated 39

40 Key Points Viable Affordability decisions yield project achievements 40 © 2012 Copyright Galorath Incorporated 40

41 © 2012 Copyright Galorath Incorporated 41 Affordability Process Step 1. Procure Key Performance Parameters that are inviolate Step 2. Identify Affordability Goals & Figures of Merit (development, life cycle, payback, ROI, NPV, kill ratio, Budget constraints, etc.) Step 3. Gather Requirements, Features, Performance Step 4. Define Baseline Alternatives Step 5. Perform Technical Design Analysis for Each Alternative Step 6. Perform Cost Schedule Analysis of Each Alternative Step 7. Assess Benefits Based on Figures of Merit Step 8. Perform Probabilistic Risk Analysis Step 9. Assess Alternatives & Select Optimal Alternative Step 10. Document Analysis and Lessons Learned

42 Requirements And Features Analysis Preliminary Design Process Human Factors Security Reliability Availability Survivability Supportability Testability Producibility Reuse Transportability Other Factors Alternative 1 Alternative 2 Alternative 3 Alternative 4 SEER Assessment Performance Schedule Risk Assessment Life Cycle Cost Selection Process OK? Optimized? Yes Cost Performance Schedule Risk Bottoms Up Estimation as Required NoIterate Should Cost: Trade Study Flow © 2012 Copyright Galorath Incorporated 42

43 © 2012 Copyright Galorath Incorporated 43 Manual Estimates: Human Reasons For Error (Metrics Can Help) Manual Task estimates yield SIGNIFICANT error without ranges Desire for credibility motivates overestimate behavior (80% probability?) So must spend all the time to be reliable Best practice approach force 50% probability & have buffer for overruns Technical pride sometimes causes underestimates 43

44 Balancing Resources & Schedule Is A Best Practice For a given Size, Complexity and Technology Impossible Minimum Time To Complete (Effort Increases to Reduce Schedule ) Work Expands To Fill Time (Effort Increases due to lack of pressure ) Inefficient Minimum TimeOptimal Effort (Lower Effort for Longer Schedule) Calendar Time Effort Months Effort Increase due to Longer Schedule © 2012 Copyright Galorath Incorporated 44

45 Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs) © 2012 Copyright Galorath Incorporated 45

46 Understanding & Tracking Defects, Growth And Other Metrics Health and Status Indicator shows status and trends from the previous snapshot Including Size Growth and Defect Discovery/Removal Rate yellowUser defined control limits to control the transition between red-yellow-green Track defect discovery and removal rates against expected rates Increased defect reporting rate shows a worsening trend Track software size growth © 2012 Copyright Galorath Incorporated 46

47 © 2012 Copyright Galorath Incorporated 47 Goal Question Metric Approach Best Practice Combine goal-orientation bottoms up, decision-support & other operational management techniques to decide to bring an umbrella is decision support Goal Question Development Contractors Metric Organizational Goal Question Development Organizations Metric

48 © 2012 Copyright Galorath Incorporated 48 Reasons Many Dont Want To Provide Data They could be proven wrong It could be used against them Data often doesnt exist Even if processes dictate data requirements If it exists, it may not be clean It may give away corporate productivity & bid strategy

49 © 2012 Copyright Galorath Incorporated 49 Data Must Be Used With Caution Run sanity checks on data A million lines of code cant be developed in 3 months Ongoing issue between our statisticians and engineers Some Statisticians claim.. That is what the data says, so it must be right Sometimes even if it is obviously wrong

50 © 2012 Copyright Galorath Incorporated 50 Data Doesnt Have To Be Perfect To Be Useful: But Is Has To Be Viable 80 Calories per serving 2.5 Servings per can 4 Ounces, Condensed, 8 Ounces With Water

51 You Have An Estimate … Now What? © 2012 Copyright Galorath Incorporated 51

52 © 2012 Copyright Galorath Incorporated 52 Correlation does not imply causation Just because two data points may sit side by side doesnt mean they are the same or will have the same outcome Casual analysis is a recognized error in medicine Perhaps ??? The Error of Causal Analysis Creating a False Association Tumor Can Cause Headache Headache doesnt mean a tumor

53 Use Historical Measurement to Evaluate Your Estimate! Its easy to dig deeper and deeper to justify an estimate! © 2012 Copyright Galorath Incorporated 53

54 © 2012 Copyright Galorath Incorporated 54 Estimation Best Practices Decide Why You Want An Estimate Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity Have A Documented, Repeatable Estimation Process Make The Estimating Process As Simple As Possible; But No Simpler Be Proactive: The Process Is Important, The Tools Go Along With The Process Get Buy-in From Program Managers Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them Tie The Estimate To The Plan

55 © 2012 Copyright Galorath Incorporated 55 Estimation Best Practices 2 Evaluate Total Ownership Cost; Not Just Development Estimate A Range And Pick A Point For The Plan Re-estimate The Program When It Changes Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale Keep A History: Start An Enterprise Database NOW… Business Case: Evaluate ROI In Addition To Costs Convert Expert Spreadsheets Into A Common Language

56 © 2012 Copyright Galorath Incorporated 56 Estimation Best Practices 3 Track Progress Vs. Estimate Throughout The Life Cycle Estimate Schedule As Well As Effort (Cost) For Complete Picture Tie The Business Case Into The Estimating Process Attack Non-productive Rework As Part Of The Process

57 © 2012 Copyright Galorath Incorporated 57 Estimation Best Practices 4 Have clear definitions: What does complete mean? What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.) Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.) Measure what you care about Estimating & tracking rework can help control costs Dont ignore IT infrastructure and IT services costs Tracking defect sources can go along with the process

58 © 2012 Copyright Galorath Incorporated 58 Conclusions Cost estimation and analysis are VITAL core processes Best practices ferret out what the cost is REALLY anticipated to be Risk and uncertainty must be taken into account Best practice project management understands the difference and acts to reduce uncertainty, or convert it to risk Applying affordability analysis to the business case yields the best value

59 © 2012 Copyright Galorath Incorporated 59 Additional Information Dan on estimating BLOG: Phone: x614

Download ppt "2012 Australian Cost Conference Keynote Cost Estimation: A Key Component of Affordable Program Success Dan Galorath Copyright 2012."

Similar presentations

Ads by Google