Presentation on theme: "Dan Galorath firstname.lastname@example.org 2012 Australian Cost Conference Keynote Cost Estimation: A Key Component of Affordable Program Success Cost estimation."— Presentation transcript:
1Dan Galorath email@example.com 2012 Australian Cost Conference Keynote Cost Estimation: A Key Component of Affordable Program SuccessCost estimation is as important as other key technical and performance variables. But unfortunately cost isn’t always viewed as equally important.. Many view the mission as the most important, regardless of the cost. Others view the technology as paramount, again without regard to cost. Still others point to cost overruns and blame them on the cost estimate itself without regard to process, risk and technical changes.Yet cost estimation is an important component of program planning and control, providing the basis for procurement decisions, affordability analysis and tradeoffs, and documentation of a planned program.Viable cost estimation and analysis are best achieved with repeatable processes and with a direct link to the program planning and control.Affordability analysis as part of decision making may be the biggest edge of the decade for both IT and mission critical defense applications.In a defense context affordability as “should cost” and “will cost” strive to replace past cost / performance failure due to inflexible user requirements or over-specified contractor requirements with realistic trades of cost, schedule, and other key performance parameters (KPPs).And of course, risk analysis is a key component of cost analysis. A single point estimate just won’t do for informed decision making.New developments and upgrades / new buys can benefit from viable cost estimating and analysis.of a risk adjusted Total Cost of Ownership, return on investment, consistency with long-range strategy, and against risk and key technical and performance parameters.This talk will discuss these cost estimation and analysis issues along with repeatable cost analysis process, and making cost analysis a key part of the solution.Dan GalorathCopyright 2012 Galorath Incorporated
3Using Best Practices Are A Best Practice Themselves “Method or technique that has consistently shown results superior to those achieved with other means, and that is used as a benchmark”“Best" practice can evolve to become better as improvements are discoveredProcessesToolsPeople
4Optimism from cognitive biases & organizational pressures Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR)Problem:Humans seem hardwired to be optimistsOptimism from cognitive biases & organizational pressuresExaggerate talents & degree of controlAttribute negative consequences to external factorsAnchoring (relying too heavily on one piece of information) magnifies optimismMost pronounced for new initiativesBest practice: Temper with “outside view”Supplement traditional forecasting w/ statistical parametricsDon’t remove optimism, but balance optimism & realism
5While Optimism Needs Tempering, So Does Short Sightedness (Source Northrop)
6Cost is a key project performance parameter Key PointsCost is a key project performance parameterNotes
7Cost Overruns Are Everywhere “GAO: Staggering cost overruns dwarf modest improvements in Defense acquisition”R&D costs of weapons programs increased 42% over original estimatesAverage delay 22 months in delivering initial capabilitiesEvolving technical requirementsShortage of qualified government staff to manage"Every dollar of cost growth on a DoD weapon system represents a lost opportunity to pay for another national priority"“The Collins Class submarine program: Murphy was an optimist”“BG Group shares hit by $5.4B cost overruns on Australian Liquified Natural Gas project”Nothing better illustrates the chronic problems than the saga of the Collins Class submarines, now in its twenty-fifth year. Astutely analysed in Lessons from Australia’s Collins Submarine Program (Schank et al. 2011), this was a project which showed Mr Murphy, of the eponymous law, to be an incorrigible optimist, at least as far as major defence ventures are concerned.Yet it would be wrong to blame bad luck for the Collins’ difficulties. Rather, from the start, almost everything that could be done wrong was done wrong. As the Coles review puts it, ‘the problems originate from the very beginning of the program when, perhaps without fully appreciating the potential consequences, the Commonwealth embarked on the acquisition of a submarine which, for good reason, is quite unlike any other in the world’. Many years later, Coles concludes, we are still in a situation where ‘despite the fact that virtually all senior people we spoke to were clear that the Collins Class capability is “strategic” for Australia, there is no clear or shared public understanding of why this is a strategic capability nor of the implications this has for sustainability.’As for acute breakdowns, those too have been in abundant supply, with the most recent being in September 2010 when the Chief of Navy imposed an ‘operational pause’ on the seaworthiness of the ‘amphibious landing platform’ HMAS Manoora,3 causing a collapse in Australia’s amphibious ship capability. That collapse, which would have made it impossible for the ADF to respond promptly to a major disturbance in our archipelagic region, is thoroughly diagnosed in the review undertaken by Paul Rizzo. The causes, he finds, include ‘poor whole-of-life asset management, organisational complexity and blurred accountabilities, inadequate risk management, poor compliance and assurance, a “hollowed-out” Navy engineering function, resource shortages in the System Program Office in DMO [the Defence Materiel Organisation], and a culture that places the short-term operational mission above the need for technical integrity.’Even more disturbingly, the Rizzo review notes that these problems were ‘long-standing, well known to Defence and DMO, and the subject of many prior reports’. And there are indeed close parallels between the conclusions of the Rizzo review and those of the 2007 Board of Inquiry into the Sea King accident at Nias Island, which concluded that the tragedy causing the death of nine ADF members during the response to the Indian Ocean tsunami was not an ‘isolated random event’ but due to ‘systemic failings across the ADF and parts of the Defence organization’ (Royal Australian Navy 2007)
8Software Is A Key Risk Item In Weapons Systems Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to be drastically underused… GAOGAO identified 42 programs at risk for cost & schedulemilitary requirements changessoftware development challengesworkforce issuesNational Institute of Standards and Technology (NIST)Software defects cost nearly $60 Billion Annually80% of development costs involve identifying and correcting defectsSoftware, not Hardware or technology readiness levels were called out
9Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR) Case Study: Levi Strauss$5M ERP deployment contractedRisks seemed smallDifficulty interfacing with customer’s systemsHad to shut down productionUnable to fill orders for 3 weeks$192.5M charge against earnings on a $5M IT project failure“IT projects touch so many aspects of organization they pose a new singular risk”
10An ROI Analysis of A New System: Should We Fund This? Can we do better?Will stakeholders tolerate a loss for 3 years?What is the risk?
11Software & IT Systems Are About Business Value Total Ownership CostDevelopmentMaintenanceIT Infrastructure & ServicesTechnical DebtValueReturn on InvestmentInternal Rate of ReturnNet Present Value“Software economics should provide methods for analyzing the choices software projects must make.” Leon LevyBusiness economics should provide methods for analyzing choices in which projects to fund
12Potentially 80% of Projects Don’t Return Adequate Value Most projects cost more than they return, Mercer Consulting:“When the true costs are added up, as many as 80% of technology projects actually cost more than they return. It is not done intentionally but the costs are always underestimated and the benefits are always overestimated.” Dosani, 2001
13IT Has Similar Failures Cutter Consortium Software Project Survey:62% overran original schedule by more than 50%64% more than 50% over budget70% had critical product quality defects after releaseStandish Group CHAOS Report46% challenged19% failed35% successful“ Fully one in six of the projects we studied was a black swan, with a cost overrun of 200% on average, and a schedule overrun of almost 70%”~$875 billion spent on IT~$300 billion spent on IT projects~$57 billion wasted annually
14Using Gates and Refining Estimates is a Best Practice (Adapted from K Using Gates and Refining Estimates is a Best Practice (Adapted from K. Aguanno)Gate 1Gate 2Gate 3Gate 4GreatIdeaOpportunity AnalysisPreliminary Business CaseCommitted Business CaseAchieveBusiness CaseRiskInvestment in time and moneyConceptMarketing AnalysisFeasibility StudyPilot or Proof of ConceptFull Execution or Deployment-Determine customer acceptance-Interview focus groups, etc.- Design solution- Estimate cost / schedule- Analyze risk- Determine feasibility / ROIBuild solution- DeployAchievebusiness caseCapture lessons learnedIncluding estimating- Describeidea &Possiblebenefits- Validate & commit to design & approachRevised estimates & scheduleRisk reductionBaselined plan
15Causes of Project Failure Source: POST Report on UK Government IT Projects Lack of a clear link between the project and the organisation’s key strategic priorities, including agreed measures of success.1.Lack of clear senior management and ministerial ownership and leadership2.Lack of effective engagement with Stakeholders3.Lack of skills and proven approach to project management and risk management4.Lack of understanding of and contact with the supply industry at senior levels within the organisation5.Evaluation of proposals driven by initial price rather than long-term value for money (especially securing the delivery of business benefits)6.Too little attention to breaking development and implementation into manageable steps7.Inadequate resources and skill to deliver the total delivery portfolio8.Source: POST Report on UK Government IT Projects
16US Better Buying Power Initiatives June 28, 2010 MandateSeptember 14, 2010 GuidanceNovember 3, 2010 ImplementationFive Specific Areas of Concern:Target Affordability and Control Cost GrowthReduce Non-Productive Processes and BureaucracyIncentivize Productivity and Innovation in IndustryPromote Real CompetitionImprove Tradecraft in Services Acquisition
17Affordability Initiatives With “Should Cost” and “Will Cost” -Cost Initiatives (Applied practices& improvements)Will CostPerformance=Should Cost Performance
18Characteristics of a Successful Should Cost Review (Source: AT Kearney)
19Cost Estimation with repeatable Process is best practice Key PointsCost Estimation with repeatable Process is best practiceNotes
20An Estimate Defined Estimates more precise with progress An estimate is the most knowledgeable statement you can make at a particular point in time regarding:Effort / CostScheduleStaffingRiskReliabilityEstimates more precise with progressA WELL FORMED ESTIMATE IS A DISTRIBUTION
21Viable Estimation Is Critical Estimating is critical for all kinds of systemsYet many treat is as a second rate processEveryone estimates…. Just most get it wrong and don’t have a processHaving a repeatable estimation process is critical to both estimating AND to successful projectsEstimation and measurement go hand in handEstimation, planning and control processes can make the difference between project success and project failure. Unfortunately, estimation is often considered unimportant by the engineering community, who may just want to make the best, fastest, etc. without considering affordability. This paper covers estimation best practices, estimation process maturity, and examples of estimation, planning and control done right. Estimation process maturity -- what it is, how to apply it to your programs and how to seek the appropriate level for your organization -- will be discussed, along with the Return on Investment of such practices.
22Estimation Methods 1 of 2 Model Category Description Advantages LimitationsGuessingOff the cuff estimatesQuickCan obtain any answer desiredNo Basis or substantiationNo ProcessUsually WrongAnalogyCompare project with past similar projects.Estimates are based on actual experience.Truly similar projects must existExpert JudgmentConsult with one or more experts.Little or no historical data is needed; good for new or unique projects.Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent.Top Down EstimationA hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component.Provides an estimate linked to requirements and allows common libraries to size lower level components.Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation.
23Estimation Methods 2 of 2 Model Category Description Advantages LimitationsBottoms Up EstimationDivide the problem into the lowest items. Estimate each item… sum the parts.Complete WBS can be verified.The whole is generally bigger than the sum of the parts.Costs occur in items that are not considered in the WBS.Design To CostUses expert judgment to determine how much functionality can be provided for given budget.Easy to get under stakeholder number.Little or no engineering basis.Simple CER’sEquation with one or more unknowns that provides cost / schedule estimate.Some basis in data.Simple relationships may not tell the whole story.Historical data may not tell the whole story.Comprehensive Parametric ModelsPerform overall estimate using design parameters and mathematical algorithms.Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable.Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation.
24“Best Value” Data Needs (adapted from Boeing Value Front) Customer Need Priorities(Decision Analysis)Models & MethodsCustomerDesirabilityAttribute ValueAttribute Value DistributionUncertaintySimulationO/SDevProdTOC Cost-Risk
25Functional Focus Example: Ladies Purse Function ………………………………………... Hold stuffCost…………………………………………… $400 at NordstromWhat else will perform the function?Paper bag - Cost = $0.05Go to plastic bag for more durabilityCost = $0.10Add color………………………………………..Cost = $0.15Add strap.……………………………………….Cost = $0.25Misses one component of customer satisfaction
26“Far Out” Higher TRL Level Estimation Goal: Better Cost For Highly Advanced Space Missions (15-20 Years in the Future)Proposed Hyperspectral ImagingSatellite predicted fielding: 2016Critical items at less than TRL 4…Like asking Edison in 1876 “How much longer for the light bulb?”“Hard to say”In 1879, once he had found a workable carbon filament, “How much will a production version of the light bulb cost to develop and produce Tom?”Then a TRL 4 questionThis capability would be of interest to:Military space asset plannersGovernment agenciesCommercial satellite producersAdvanced concept designers
27Black Swans (Unknowns) The match is unlikely to ever be perfect because some projects are affected by “unknown unknowns,” also called Black Swans, that is, events that are essentially unpredictable (e.g., a severe worldwide credit squeeze)ScoreOverrun %ScoreOverrun %Score
28Dealing With the “Problem of Assumptions” Assumptions are essential but…Incorrect assumptions can drive an estimate to uselessnessUse an assumption verification process1. Identify assumptions2. Rank order assumptions based on estimate impact3. Identify high ranking assumptions that are risky& quantify what happens if those assumptions change4. Clarify high ranking, high risk assumptions5. Adjust range of SEER inputs to describe the uncertainty in assumptionsEstimates must have assumptions defined, but…Bad assumptions should not be justification for bad estimates
29System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK “If you can’t tell me what it is, I can’t tell you what it costs.” -Mike JeffersWithout a good over-arching system description, the program is just a group of parts or subsystems. A system description, like that in a CARD, would help observers understand what the system does, how it works, program risks, and any operating parameters.“If you can tell me the range of what it might be, I can tell you the range of cost, schedule & probability.” -Dan Galorath
30Uncertainty in the Cost Depends On Uncertainty of the Project Itself SEER includes uncertainty in its estimatesWithin 10%Even though the entire project may be highly uncertain, tasks to the next gate should be estimatible within 10%.
32Estimating Accuracy Trumpet Range vs. Point Estimates (Source US Army)Range of Risk & UncertaintyEstimating Accuracy TrumpetTechnical and Program MaturityROM -30% to +75%Parametric -10% to +20%Analogy -15% to +30%Engineering-5% to +15%Actual-3% to +10%Target CostPoint estimate is most likely within range estimate with higher potential for cost increaseRange estimate provides a degree of risk and uncertaintyUpper boundLower bound+75%-30%BAMaterielSolutionAnalysisSystems AcquisitionSustainmentTechnology DevelopmentProduction & DeploymentOperations &SupportEngineering andManufacturing DevelopmentCPre-Systems Acquisition
33Understand the risk before you commit! Firm Fixed Price?Feel lucky?What is likely to happenUnderstand the risk before you commit!33
34Dealing With Early Estimates Give a range: but they will hear the bottom of the rangeGive a high probability number:Will still be low in some cases and may be high in many cases but consider it a probably “not to exceed”Sticker shock may be a problemGive a category rather than an number: e.g over $10m Cat 1; over $5m Cat 2; over $1m Cat 3, etc.Stakeholders always remember the first number even when told it is preliminary.Developers will be optimistic by nature unless the process is tempered.
35Potential Accuracy at Various Gates (Adapted From Canada Treasury) Concept / Approach+- 10% to next gate+_50 with Analogies & kbasesBusiness Case+_ 40%Analogies, Kbases & some specificsProject Charter & Plan+_25%Detailed Plan & functional specs+_ 15%SEER +_10% if good inputsConstruction / Deployment+- 0%Post-Deployment+-0%NOTE Should understand if full functionality was delivered or if schedule/cost relief based on deferred functionalityWhile Galorath states SEER is Within 10%, many organizations report much closer3 The Gating FrameworkOverview of the gating modelThis chapter describes the gating model recommended by TBS as a process to ensure that IT-enabled projects are on course for success. The full gating model defines seven gates. The purpose and issues associated with each gate, as well as types of reviews typical for the gate, are summarized in table format in Sections 3.1 to 3.7. In addition to the full gating model, there is also a streamlined five-gate model for projects of medium size and complexity and a light three-gate model for smaller, low-risk projects. The streamlined and light gating models combine certain gates and provide feasible alternatives if less rigorous project assessment will suffice.To give executives who are accountable for large and complex IT-enabled projects effective discipline and control, it is recommended that those projects be structured to provide for a clear, comprehensive, and objective assessment of how the project is performing against planned objectives at all stages. Key to success is ensuring that resource implications and results are visible to executives at logical predetermined checkpoints, or "gates." Gates provide the opportunity for an informed assessment of progress and issues. This permits executives to make better decisions on future plans and investments.The gating model recommended by TBS draws upon ideas from the United Kingdom's Office of Government Commerce (OGC) Gateway Process and the Province of Ontario's IT Project Gateway Review Process as well as components of the Stage-Gate Process adopted by several departments. All of these frameworks and methodologies are tried and proven. However, the TBS-recommended gating model also takes into account the decentralized and less standardized environment that exists in the federal government and the larger size of typical IT-enabled projects (compared with provincial jurisdictions and most private sector organizations). This includes the tendency in the federal government to procure project components piece by piece rather than all at once for the complete project or business service.The full gating model defines seven gates that might logically be present in every project. Its actual application, however, will depend on each case. The seven gates are:Gate 1—Strategic assessment and conceptFor confirmation of the project's objectives—both what is to be done and why—and the identification of key stakeholdersGate 2—Project approachFor confirmation of how the project's objectives will be achievedGate 3—Business case and general readinessFor confirmation of funding and business outcomesGate 4—Project charter / project management plan (PMP)For confirmation of resources, support, and governanceGate 5—Detailed project plan and functional specificationsFor confirmation of readiness to proceed with constructionGate 6—Construction complete and deployment readinessFor confirmation of readiness to deploy for both business and IT domainsGate 7—Post-implementation reviewA post-mortem and final step to gather lessons learned.The seven-gate model described in this handbook, along with other gate models that may be in use, follow similar principles. Gates are created such that at each successive gate, the project becomes more precisely defined, and uncertainties and risks become clearer and are resolved or mitigated. The project is also expected to have made certain accomplishments, indicated by project progress documents that permit an assessment to be made of the project's state and its readiness to proceed to the next gate. In addition to reviews associated with a gate, health check reviews or workshop reviews on particular issues may also occur at other times. The figure in Appendix A, "Gating," shows the positioning of gates relative to the classic SDLC (systems development life cycle). The choice of health check and workshop reviews in Appendix A are merely examples, whereas the gate reviews are shown where they would typically occur.While the SDLC follows a traditional waterfall methodology, the positioning of gates can be adapted to various methodologies, overlapping phases, and multiple-release projects. (A waterfall methodology is one in which the entire scope of the project passes through each phase before work begins on the next step. This is the opposite of the iterative methodology in which an initial solution that meets only part of the requirement is developed and implemented, followed by subsequent cycles of development and implementation.)Deciding which gates to use for a project is important. In departments where there is an executive-level oversight group for project or investment management, this group may make a decision based on its assessment of the project's risk or complexity. In other departments, the decision may be left to the project sponsor.The best practice is that all projects go through all seven gates. If a less rigorous course is considered acceptable, then streamlined gating and light gating are viable options (see the following figures of streamlined gating and light gating). Independent reviews are performed at gates where an external opinion is felt to be beneficial. On large and/or long-term projects, it is recommended that at least one independent full review or health check review should occur each year. The Review Topics for Enquiry supplement to this handbook provides the reviewer with a comprehensive summary of issues for consideration, depending on the circumstances at hand. Lines of examination can be selected as appropriate to the applicable gate and review type (see Section 2.3, "Types of Reviews").In the tables that follow describing each of the gates, percentages indicate the accuracy of project estimates that reviewers should normally expect to find. At each gate, two estimates are examined: an estimate for the entire project and an estimate of the work required between the most recently completed gate and the next gate. The first estimate forecasts the total project effort. Normally, this estimate would change from an extremely rough approximation during the early gates when the project is still a concept, to an increasingly accurate estimate at later gates, until it finally reduces to an estimate of ±15 per cent at Gate 5, just before construction starts.The second estimate forecasts the work that needs to be done before the next gate. Since this is work that is about to begin immediately and most issues should be known, the estimate should be accurate (±10 per cent); this requirement appears in the "Supporting items" section of the tables.It is important not to confuse these two different estimates—one is for the project overall, and the other is for the work necessary to arrive at the next gate.Here are three project phase and gating models:Gating Model 1: Full gating for very large and highly complex projects:Gating Model 1 - Text versionGating Model 2: Streamlined gating for projects of medium size, risk, and complexity:Gating Model 2 - Text versionGating Model 3: Light gating for small, low-risk projects with little complexity:Gating Model 3 - Text version3.1 Gate 1 Review—Strategic assessment and conceptPurpose: To answer the key questions "What do we want to do?" and "Why?" The objectives at this early stage are to test the wisdom and appropriateness of the proposed undertaking, and to ensure that key stakeholders are identified, and that everyone understands what is to be done and why. A half- to one-day workshop session is typical, preceded by reading of any available early project documents. The Gate 1 review seeks to arm the review sponsor with considerations that should be addressed before the next phase of the project and, in some cases, may actually dictate going back to the drawing board before proceeding further.Review issues Validation of the rationale for the projectConfirmation that underlying fundamentals make senseAssessment that the project is doable as proposedWhy To eliminate ideas that do not make sense or will prove to be impossible to executeCore review items for this gate Wisdom and appropriateness of proposed undertakingArticulation of the business problem and validity of the business imperativeDefinition and boundaries of scopeDefinition and measures of successDegree of common understanding about the proposal among parties involvedIdentification of stakeholders and extent of support and commitment for the initiativeConfirmation that the project makes sense in the context of the departmental project portfolio and federal government prioritiesSupporting item(s) Plan and estimate (±10%) for tasks and level of effort to next project gateTypical input to the review The project group provides reviewers with an understanding of why the project is being proposed, what it is intended to accomplish, and how it is defined. Supporting information to position the full context of the proposal might include reference to departmental reports and plans as well as an overview of the department's main business lines.The following areas need to be addressed:The concept and imperative of the project, including identification of the project sponsor, the business problem statement in the context of the overall business strategy, the broad scope of the project, expected general business outcomes and indicators of success, key stakeholders, general business risks, approximate sizing of the project, and critical success factors.The alignment of the project proposal with departmental Program Activity Architecture, program(s) delivery, departmental plans and priorities, broader federal government or cross-departmental goals, as appropriate, and positioning of the project in the context of the departmental information management and IT portfolio (including reference to departmental architectures). The positioning might also reference TBS-sponsored shared or common services and cluster initiatives, if relevant.Review format Workshop review (conducting a few targeted interviews may be necessary)3.2 Gate 2 Review—Project approachPurpose: To confirm that the approach to address the business problem or opportunity is wise by testing its feasibility and appropriateness. As in the Gate 1 review, this gate is not intended to be an overly burdensome undertaking and can often be performed in a one-day workshop session. The project group is expected to provide information on the approach envisaged for the undertaking—a discussion that should flow logically from the focus on the business problem and alignment issues of the Gate 1 review.Review issues Validation of the wisdom and feasibility of the proposed approach to the projectWhy Experience and studies show that unwise decisions on project approach have frequently proven to be a major contributing factor to project failureCore review items for this gate Reconfirmation of underlying business wisdom, precision of goals, commitment of stakeholders, and how expected business outcomes will be measuredProject classification (e.g., sustaining, tactical, etc.). See Appendix C for definitions of project classAssessment of key elements in the approach description document, including:How much redesigning will be required for the business process, model, and program delivery strategyHow business change will be decided upon and achievedPrecision of definition and scope—is the project definable and can its scope be bounded?Preliminary business modelExtent of reuse of existing assets—systems, data, business rules, procedures, and program delivery infrastructuresHow transition to new environments will occur for both the business program and the associated IT systemsMake-versus-buy considerationsEnvironment in which project will be undertakenPreview of risk assessment for the selected approachPreview of how project will be governedNote: The expected level of detail for the above items relates to what is necessary to permit an assessment of the feasibility of the approach; the focus is on whether the approach is reasonable.Supporting item(s) Reviewers would also benefit from previewing:Project packaging (order of magnitude sizing [±100%]), possible technological solutions, project staging and time frames, key roles and staffing considerations, goal alignment, architectural considerations, key assumptions, and risksProject environment (governance, relative departmental capacity, project culture and discipline, business ownership, executive engagement and capacity, supporting functions such as human resources, finance, procurement, etc.)Updated plan and estimate (±10%) for tasks and level of effort to next project gateTypical input to the review Information on project approach according to core review itemsPreliminary results from a Project Complexity and Risk Assessment (PCRA) and Organizational Project Management Capacity Assessment (OPMCA), if requiredNote that in some situations, particularly for smaller projects, Gates 1 and 2 may be combined.3.3 Gate 3 Review—Business case and general readinessPurpose: To answer the key question "How?" The most feasible project options are considered here and the preferred option recommended. The project approach is now fully articulated. A high-level project plan has been tabled, upon which preliminary costing is based. The business case should include an investment rationale and describe outcomes to be achieved, as well as their justification relative to the proposed cost. Project cost and schedule estimates for the entire project should be in the 40% range.(Ranges assume a typical development or integration project where there are many unknowns at the early stages. An infrastructure replacement project might have a much smaller estimating range.)Review issues Assurance that the business case is thorough, complete, and compellingConfirmation that the organization is ready to undertake the projectWhy To confirm that the business case is sufficiently compelling to justify, sustain, and guide the projectTo identify any shortcomings in readiness for action before approval and confirm that key identified risks can be managedCore review items for this gate Business case requirements:Clarity of business problem statementClarity and precision of project goals—must be sufficiently clear to provide focussed outcomes, and guide what is in and out of project scopeClear business justification for the project investment, including how goals can be quantified and measured, and their attainment confirmedReconfirmation of the alignment of the project with organization's goalsThoroughness of options analysis, including reasonableness of costs and benefits for each option and reasonableness of selected option Indicative cost estimate and scheduleEstimating methodology, basis for assumptions, and sensitivity analysisOrganizational readiness to undertake the project:Arrangements decided upon—Project Management Office (PMO), strategies for outcome management, performance management, and risk managementInitial project planning under way and mechanisms in placeBusiness requirements approach fully definedPreparation of environment in which to run projectAssessment of organizational capacity relative to project difficulty (OPMCA)Evidence of intent and ability to create an environment for project successSupporting item(s) Complete definition of the project:Project complexity and risk levels, scope, size, and packagingRisks, constraints, and dependencies (PCRA)Architecture and technological alignment (within federal government and department)Privacy issues (Privacy Impact Assessment, or PIA), security issues (Threat and Risk Assessment, or TRA), and other policy issuesUpdated plan and estimate (±10%) for tasks, level of effort to next project gate PCRA and OPMCATypical input to the review Business case standard, and detailed supporting analysis and documentationPCRAOPMCAReview format Workshop review; in very large or complex projects, may be a quick review3.4 Gate 4 Review—Project charter / PMPPurpose: To address business case unknowns. A complete project charter, high-level PMP, and definition of the business solution are prerequisites for this gate. Any business case unknowns should be resolved here (or a plan in place to resolve them). Project cost and schedule estimates for the entire project should be ±25%.Review issues Validation that the project charter has addressed all issues critical for a successful projectConfirmation that proper project governance, planning, and management are in placeWhy To ensure that all necessary ingredients for successful execution are in place prior to construction and implementationTo reduce the risk of having to introduce quick fixes laterCore review items for this gate Reconfirmation of business case, particularly feasibility of realizing outcomes, assumptions, constraints, and dependencies:Refinement of estimates and estimating assumptionsReconfirmation of readiness to undertake the project Completeness of the project charter:Absolute clarity of definition, and what is in and out of project scope Clarity of roles, project sponsor, stakeholders, and governanceDocumented definition of success, business outcomes and how they will be measured, and clarity of accountability for attaining the business outcomesSummary of approach, with a focus on roles, governance, and risksIdentification of risks and assessment of whether risks are contained well enough to proceedDefinition of business solution:Business architecture or modelSolution description, including high-level program design and business model; business transformation strategy and business process re-engineering strategy, if applicableHigh-level functional requirements, and technical and performance requirementsHigh-level data model and data considerationsHigh-level functional design and concept of operationsCommercial off-the-shelf options assessment, if applicableHigh-level PMP:Elaboration of the project plan (work breakdown structure) and link to estimated costs and scheduleRequirements-gathering workshopProcurement plan—Is it achievable in project time?Business change management strategy and ability to be absorbed by organizationBusiness deployment strategy, including data issuesSkeleton PMO in place, suitable project manager identified, and key project staffing initiatedSupporting item(s) Updated plan and estimate (±10%) for tasks and level of effort to next project gateTypical input to the review Updated business caseProject charter standardPreliminary PMPSolution descriptionReview format Full review for larger projects or quick review as considered appropriate for smaller, low‑risk initiatives3.5 Gate 5 Review—Detailed project plan and functional specificationsPurpose: To confirm the completeness and feasibility of the detailed project plan and definition of requirements. Decisions to go ahead with major spending commitments and to make major procurement choices (such as issuing requests for proposals or awarding contracts) take place here. All major unknowns have been sufficiently researched to provide assurance that high-impact risks have been effectively mitigated. Estimates for the entire project should be ±15%. (Note that project contingency should never fall below 10–15%).Review issues Ensure that the detailed plan provides a firm baseline for managing and tracking, and that unknowns are reduced to a minimumAssess clarity of project deliverables and accountabilityWhy To ensure that executive(s) responsible for the project have a clear basis for assessing progress and taking remedial actionTo ensure that the project is ready to proceed to construction with minimal risk of delays caused by inattention to essential detailsCore review items for this gate Reconfirmation of business case and project charter Detailed management plan:Detailed project scope—What is in scope and what is out?Project dependencies specified with appropriate commitments documented, including an assessment of impact and riskArchitectural plans and decisions specified (architectural review completed)Business change management and detailed business deployment plans, including data migration and conversion, training, and transition plansProject organizational structure and resource requirements fully definedDetailed work breakdown structureTraceability matrix in place for requirementsDetailed list of deliverables and high-level acceptance planPreliminary implementation and system migration plans, including release management planDetailed schedule, substantive budget estimates (with assumptions, contingencies)Tracking, control, and status reporting set-upDetailed outcomes measurement planUpdated risk assessment, PCRA, and OPMCAFirm costs, refined estimates, and estimating assumptions Management and key position staffing (full PMO tool set in place) High-level functional requirements (Requirements Traceability Matrix) and design; perhaps proof of concepts and design workshops to ensure solution in principle Business change management and deployment planning Architectural, technological, and design decisions taken Procurement plans and Request for Proposal progress or documents Security certification, privacy plans. TRAs, PIAs performed; mitigations stated Acceptance and outcomes measurement planTypical input to the review Complete detailed project planAll sign-offs on procurement requirements, TRAs, PIAs, etc., as required to allow construction to proceedReview format Quick or full review, depending on project3.6 Gate 6 Review—Construction complete and deployment readinessPurpose: To verify that the system under development is ready for implementation and that the project is fully prepared for a successful deployment. This gate represents a major point of approval for business readiness. There may be only one or a number of sub-gates and sub-gate reviews related to construction and deployment. (Other gateway approaches reviewed tended to treat construction or construction and deployment as one gate. This is perhaps because the projects under consideration were small or because there is an assumption that construction and deployment are well understood and have relatively low risk.) Based on the given situation and the degree of risk, the department should decide on the number, timing, and focus area for intermediate Gate 6 reviews during construction and deployment. For example, depending on the project structure, a gate could be established at the completion of a major development release or at the completion of a major rollout and deployment to a specified set of users. (See "Notes on Gate 6" below.)Review issues Verification that construction is complete with user acceptance testing and migration to production firmly based on meeting acceptance criteria that support proceeding with deploymentConfirm that deployment teams have been established and are prepared to manage a smooth transition Why To ensure the project is ready to proceed with deployment, and that system migration plans and ongoing support are in placeCore review items for this gate Reconfirmation that the project is aligned with departmental goals, the business case is valid, and expected outcomes will occurConstruction complete and deliverables, including all documentation, acceptedSystem migrated to production, and user acceptance testing completeBusiness change management, deployment, training, data migration, and conversion plans completeSystem migration plan validatedOngoing support and service management plans in placeVulnerability assessment completeOverall business and project readinessSupporting item(s) Updated plan and estimate (±10%) for tasks and level of effort to project close-outTypical input to the review All sign-offs for user acceptance, production acceptance by operations, security certification and accreditation to go into production, maintenance team acceptance of documentationDeployment, training, data migration and system migration plans, and vulnerability assessment approvedSupport and service management planDisaster recovery and business resumption plansReview format Quick or full review, depending on project size, risk, and complexityNotes on Gate 6Projects may adopt a variety of phasing approaches, and not all phasing approaches will be sequential. Some may be structured with parallel phases, making the choice of gates and the scope of gate reviews more subjective. In the case of large iterative development methodologies, gates and review points would ideally be established to ensure that development is moving toward closure—that the iterations would not continue indefinitely or until the project runs out of time and money. The focus, then, is on the sign-offs against what has been delivered and on the clear agreement on what remains to be done. Examples of intermediate gates within the purview of Gate 6 include:Construction release completion—In a project that is structured to have multiple major releases, it is recommended that a gate be established at the completion of one or more of these releases.Mid-phase health check—A health check could be established in the middle of a project phase even if no major deliverables have been completed or decision points reached. The decision to do so might be based on some combination of dollars spent (e.g., $25 million), time elapsed (e.g., one year), and rate of expenditure (e.g., $2 million per month).Construction and deployment readiness—In many projects, deployment activities and construction actually occur in parallel. In some cases, deployment occurs after the completion of construction. Depending on the nature and size of the project, a gate might be established to create a decision point about whether or not to deploy.Pilot deployment and full deployment readiness—In some projects, a pilot deployment occurs immediately following the construction phase as a basis for deciding whether the system is ready for general deployment. This might be an appropriate point at which to establish a gate, depending on the size and nature of the project.3.7 Gate 7 Review—Post-implementation reviewPurpose: To confirm completion, assess the extent to which the project has achieved its goals, and provide an assessment of value for money. This gate typically occurs approximately six months following project completion. A review at this point can also catalogue the lessons learned during the project—those identified by the project group and those captured by independent reviewers.Review issues Verify that the project was completed as planned and that the expected business outcomes were actually realizedAssess the degree of success for the transition to an ongoing serviceWhy To confirm success from a project delivery and business perspectiveTo determine what lessons might benefit the department and the broader community in future undertakingsCore review items for this gate Project delivery measured against original objectivesConfirmation of the archiving of information and deliverables, as applicableKnowledge transfer and transition to successful serviceCompletion of contractual obligationsValidation of business outcomesCapture of lessons learned, including review processProject close-out report completedTypical input to the review Project close-out reportBusiness outcomes measurement planContract acceptance reportsProject lessons learnedReview format Workshop to full review, depending on projectIn some cases, it may be premature to definitively determine the achievement of business outcomes during this gate. In that case, a plan might be developed to address further follow-up on the core items.An important area for assessment is the effectiveness of the independent review itself and its positive and negative impact on the outcome of the project. It is recommended that lessons learned about project management and independent reviews be fed back to TBS CIOB for continuous improvement.Estimate Accuracy Is a Function ofInput Information Quality.Estimates Can Be Much Closer than ShownIF Data Is Available.
36Converting Uncertainty to Risk Is A Best Practice Many treat as synonymous but…In risk situations, probabilities assigned based on dataIn uncertainty situations, we may assign probabilitiesBut there is no data to back them upIf probability of rain tomorrow is 50%, that’s a riskWe have historical data & scientific analyses about rain which make it reasonable to estimate the probabilityIf we say Probability of success in harnessing nuclear fusion for routine energy production within the next ten years is 50%, that’s just a guess about uncertainty—evidence for the assignment is lacking or very weakIf you toss this thumbtack, what is the probability it will land this way instead of on its back?
37GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them This chapter discusses a 1972 GAO report on cost estimatingWe reported that cost estimates were understated and causing unexpected cost growthMany of the factors causing this problem are still relevant todayWe also discuss a 12 step process for producing high quality cost estimates
38Generalized 10 Step System Estimation Process 2011 Establish Estimate ScopeTrack Project Throughout DevelopmentEstablish Technical Baseline, Ground Rules, AssumptionsDocument Estimates and Lessons LearnedGenerate a Project PlanRefine Technical Baseline Into Estimable ComponentsValidate Business Case Costs & Benefits (go / no go)This software estimation process is described in Software Sizing, Estimation and Risk Management by Dan Galorath and Michael Evans.Collect data / estimation inputsQuantify Risks and Risk AnalysisEstimate Baseline Cost, Schedule, Affordability Value38
39Contractors At Least Level 3 Would Be Acceptable Informal or no estimatingManual effort estimating without a processLevel 1Direct Task EstimationSpreadsheetsAd Hoc ProcessLevel 2Formal Sizing (e.g. function points)Simple model (Size * Productivity) or informal SEER UseSome measurement & analysisInformal ProcessLevel 3Formal SizingRobust Parametric estimation (SEER)Estimate vs. actual captureFormalized Multiple Estimate ProcessRigorous measurement & analysisParametric planning & ControlRisk ManagementRepeatable processLevel 4Formal sizingRobust parametric estimating (SEER)Parametric estimation with tracking & controlProcess improvement via lessons learnedLevel 5Continuous process improvementWhy should we care? Maturity is related to estimate viability… Better estimation process more likely to be successful in execution
41Affordability Process Step 1. Procure Key Performance Parameters that are inviolateStep 2. Identify Affordability Goals & Figures of Merit (development, life cycle, payback, ROI, NPV, kill ratio, Budget constraints, etc.)Step 3. Gather Requirements, Features, PerformanceStep 4. Define Baseline AlternativesStep 5. Perform Technical Design Analysis for Each AlternativeStep 6. Perform Cost Schedule Analysis of Each AlternativeStep 7. Assess Benefits Based on Figures of MeritStep 8. Perform Probabilistic Risk AnalysisStep 9. Assess Alternatives & Select Optimal AlternativeStep 10. Document Analysis and Lessons LearnedStep 1. Procure Key Performance Parameters that are inviolateStep 2. Identify Affordability Goals & Figures of merit (development, life cycle, payback, ROI, NPV, kill ratio, Budget constraints, etc.)Step 3. Gather Requirements, Features, performanceStep 4. Define baseline alternativesStep 5. Perform technical Design Analysis for each alternativeStep 6. Perform cost schedule analysis of each alternativeStep 7. Assess benefits based on figures of meritStep 8. Perform Probabilistic Risk AnalysisStep 9. Assess Alternatives & Select optimal AlternativeStep 10. Document analysis and lessons learned
42Requirements And Features Should Cost: Trade Study FlowRequirements And FeaturesSelectionProcessAnalysisAlternative 1Alternative 2Alternative 3Alternative 4SEERAssessment4532PreliminaryDesignProcessPerformanceScheduleRisk AssessmentLife Cycle Cost1Other FactorsHuman FactorsSecurityReliabilityAvailabilitySurvivabilitySupportabilityTestabilityProducibilityReuseTransportabilityIterateNoOK?Optimized?YesCostPerformanceScheduleRiskBottoms Up Estimation as Required
43Manual Estimates: Human Reasons For Error (Metrics Can Help) Manual Task estimates yield SIGNIFICANT error without rangesDesire for “credibility” motivates overestimate behavior (80% probability?)So must spend all the time to be “reliable”Best practice approach force 50% probability & have “buffer” for overrunsTechnical pride sometimes causes underestimates
44Balancing Resources & Schedule Is A Best Practice For a given Size, Complexity and TechnologyCalendar TimeEffort MonthsImpossibleMinimum TimeTo Complete(Effort Increases to Reduce Schedule)Work Expands To Fill Time(Effort Increasesdue to lack of pressure)InefficientEffort Increase due to Longer ScheduleMinimum TimeOptimal Effort(Lower Effort for Longer Schedule)Brooks’ Law describes the time-effort tradeoff curve for a given Size and Technology.Paul Masson’s Law establishes the minimum time estimate and eliminates the “impossible” portion of the time-effort curve.Parkinson’s Law establishes the optimal effort estimate and discourages the “inefficient” portion of the time-effort curve.
45Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs) 48121620Schedule Probability Example Application 1ProbabilityTime (calendar months)1%10%20%30%40%50%60%70%80%90%99%18003600540072009000Effort (person-hours)1%10%20%30%40%50%60%70%80%90%99%Effort Probability Example Application 1Probability1224364860Defects Probability Example Application 1ProbabilityDefects (count)1%10%20%30%40%50%60%70%80%90%99%Point out that SEER-SEM can produce probability charts for time, effort, cost, and defects.
46Understanding & Tracking Defects, Growth And Other Metrics Track defect discovery and removal rates against expected ratesIncreased defect reporting rate shows a worsening trendTrack software size growthHealth and Status Indicator shows status and trends from the previous snapshotIncluding Size Growth and Defect Discovery/Removal RateUser defined control limits to control the transition between red-yellow-green
47Goal Question Metric Approach Best Practice DevelopmentContractorsMetricOrganizational GoalOrganizationsCombine goal-orientation bottoms up, decision-support & other operational management techniquesto decide to bring an umbrella is decision support
48Reasons Many Don’t Want To Provide Data They could be proven wrongIt could be used against themData often doesn’t existEven if processes dictate data requirementsIf it exists, it may not be cleanIt may give away corporate productivity & bid strategy
49Data Must Be Used With Caution Run sanity checks on dataA million lines of code can’t be developed in 3 monthsOngoing issue between our statisticians and engineersSome Statisticians claim.. “That is what the data says, so it must be right”Sometimes even if it is obviously wrong
50Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable 80 Calories per serving2.5 Servings per can4 Ounces, Condensed, 8 Ounces With Water
52The Error of Causal Analysis Creating a False Association Correlation does not imply causationJust because two data points may sit side by side doesn’t mean they are the same or will have the same outcomeCasual analysis is a recognized error in medicinePerhaps ???Tumor Can Cause HeadacheHeadache doesn’t mean a tumor
53Use Historical Measurement to Evaluate Your Estimate! It’s easy to dig deeper and deeper to justify an estimate!
54Estimation Best Practices Decide Why You Want An EstimateMap Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The MaturityHave A Documented, Repeatable Estimation ProcessMake The Estimating Process As Simple As Possible; But No SimplerBe Proactive: The Process Is Important, The Tools Go Along With The Process Get Buy-in From Program ManagersHold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own ThemTie The Estimate To The Plan
55Estimation Best Practices 2 Evaluate Total Ownership Cost; Not Just DevelopmentEstimate A Range And Pick A Point For The PlanRe-estimate The Program When It ChangesAvoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain MoraleKeep A History: Start An Enterprise Database NOW…Business Case: Evaluate ROI In Addition To CostsConvert Expert Spreadsheets Into A Common Language
56Estimation Best Practices 3 Track Progress Vs. Estimate Throughout The Life CycleEstimate Schedule As Well As Effort (Cost) For Complete PictureTie The Business Case Into The Estimating ProcessAttack Non-productive Rework As Part Of The Process
57Estimation Best Practices 4 Have clear definitions: What does “complete” mean? What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.) Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.)Measure what you care aboutEstimating & tracking rework can help control costsDon’t ignore IT infrastructure and IT services costsTracking defect sources can go along with the process
58Conclusions Cost estimation and analysis are VITAL core processes Best practices ferret out what the cost is REALLY anticipated to beRisk and uncertainty must be taken into accountBest practice project management understands the difference and acts to reduce uncertainty, or convert it to riskApplying affordability analysis to the business case yields the best value
59Additional Information Dan on estimating BLOG:Phone: x614