28 Aug 2012 PRT-148  Los Angeles  Washington, D.C.  Boston  Chantilly  Huntsville  Dayton  Santa Barbara  Albuquerque  Colorado Springs  Goddard Space Flight Center  Johnson Space Center  Ogden  Patuxent River  Washington Navy Yard  Ft. Meade  Ft. Monmouth  Dahlgren  Quantico  Cleveland  Montgomery  Silver Spring  San Diego  Tampa  Tacoma  Aberdeen  Oklahoma City  Eglin AFB  San Antonio  New Orleans  Denver  Vandenberg AFB Joint Analysis of Cost and Schedule (JACS) Australian Department of Defence 2nd Cost Estimation Conference October 2012 Alfred Smith, CCEA Jennifer Kirchhoffer, CCEA

28 Aug 2012 PRT-148 Approved for Public Release2 of 35 n What is JACS? n Overview of the JACS modeling process n Key reports from a well constructed JACS model n Concluding remarks Agenda

28 Aug 2012 PRT-148 Approved for Public Release3 of 35 n Cost, schedule and risk assessments traditionally have been performed by separate teams of professionals n In recent years, it has become more common for the cost analyst to report a “risk adjusted” result as a budget recommendation rather than a point estimate n However, it appears that cost uncertainty models routinely: try to force a 70 or 80% cost result into the point estimate schedule ignore risk management team statements like “High probability this event will occur and if it does, the consequence will be severe” What is Joint Analysis of Cost and Schedule (JACS)? Joint Analysis of Cost and Schedule is a disciplined, systematic and repeatable process to integrate three critical pieces of information: Cost, Schedule, Risk

28 Aug 2012 PRT-148 Approved for Public Release4 of 35 Current Approach to Model Cost Estimating Uncertainty Cost Input, e.g., weight Cost = a + bx c Historical data points Sources of Uncertainty: Cost estimating method Cost method inputs Focus is on estimating total cost uncertainty with limited influence from duration uncertainty or potential events that may influence cost/schedule.

28 Aug 2012 PRT-148 Approved for Public Release5 of 35 n Capture schedule uncertainty on Time-Dependent Costs n Inclusion of Discrete Risks (5x5’s) Evolving Trends in Uncertainty and Risk Analysis Time-Dependent (TD) [Level of Effort - LOE] Risk 1 Risk 2 Risk n Translate 5x5 into probability of occurrence times uncertain consequence which can impact cost and/or duration of one or more tasks 1 n 2

28 Aug 2012 PRT-148 Approved for Public Release6 of 35 Project Start Project End Task Duration Integrated Risk & Uncertainty Landscape – the JACS Paradigm TD = Time-Dependent Cost, e.g. ‘marching army’ cost TI = Time-Independent Cost, e.g., Materials Duration Uncertainty Uncert Burn Rate Uncertainty TD $ = Segment Duration X Burn Rate Uncert Probability of Occurrence Risk Register Uncert TI $ Uncert TI $ TI $ Uncertainty Uncert TI $

28 Aug 2012 PRT-148 Approved for Public Release7 of 35 n What is JACS? n Overview of the JACS modeling process n Key reports from a well constructed JACS model n Concluding remarks Agenda

28 Aug 2012 PRT-148 Approved for Public Release8 of 35 The JACS Process: Develop the Analysis Schedule Risk Sched Cost Collect Sched Data Create Analysis Schedule Validate Before Continuing

28 Aug 2012 PRT-148 Approved for Public Release9 of 35 n Joint analysis of cost and schedule begins with a model of the schedule logic Serves as the backbone for the analysis Cost, risks and uncertainty are mapped into the logic to assess impacts n Project/program integrated master schedules (IMS) are unsuitable for this role They are generally too big, complex and too detailed Logic common in an IMS can be a problem for a JACS analysis (e.g., constraints) n A JACS appropriate schedule must be created from available data (including the program IMS) Typically referred to as an “analysis schedule” The Need for an Analysis Schedule

28 Aug 2012 PRT-148 Approved for Public Release10 of 35 √ Captures the major work-flows of the project IMS √ Provides insight into major cross-dependencies within or across management responsibility boundaries √ Creates a solid framework to capture cost / schedule uncertainties and discrete risk events √ Structured around management/ budget responsibility √ Allows mapping of budgeted work effort to schedule scope √ Aligns with cost/budget data √ Identifies key tasks that support major deliverables/ tracking items √ Detailed IMS step by step work items and task flows are combined while maintaining critical path logic √ Has traceability and transparency to the more detailed IMS Attributes of an Analysis Schedule

28 Aug 2012 PRT-148 Approved for Public Release11 of 35 The JACS Process: Map Costs to Schedule Tasks Risk Sched Cost Collect Sched Data Create Analysis Schedule Update Analysis Schedule Collect Cost Data Identify as TD or TI Map to Sched Activities

28 Aug 2012 PRT-148 Approved for Public Release12 of 35 n Easier to map cost model results to a schedule model rather than replicating schedule network logic in a cost model Schedule models are generally populated with throughputs, that is they don’t allow equations to estimate the cost of one task based on the cost of another or some technical characteristics n Mapping cost estimate to a schedule model is simplified by: Unifying cost (often product based) and schedule (often task based) work breakdown structures Specifying Time Dependent and Time Independent costs and their uncertainty separately Defining how the TI or TD cost is phased over the task duration Mapping of Cost to Schedule TD Cost TD Phasing Total Cost TI Cost TI Phasing

28 Aug 2012 PRT-148 Approved for Public Release13 of 35 The JACS Process: Mapping the Risk Register to Schedule Tasks Risk Sched Cost Collect Sched Data Create Analysis Schedule Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Update Analysis Schedule Collect Cost Data Identify as TD or TI Map to Sched Activities

28 Aug 2012 PRT-148 Approved for Public Release14 of 35 n Defined as: If risk event A occurs, there is a cost consequence or opportunity. The probability of A occurring is x% Often modeled as a separate task inserted into the schedule network n If there are only a few such risk events, treat as discrete what-if cases (event cost or schedule impact is either “in” or “out” of the estimate) Point estimate includes the full impact (when there are few) n If there are many such risk events, model using the Yes/No distribution (also known as the Bernoulli distribution) When there are many, there is no standard on how to treat the Point Estimate:  Include none and assess separate from the Point Estimate?  Include all (worst case scenario)?  Include the sum of the expected values?  this is a common approach The risk register should account for:  Uncertainty of the cost consequence or opportunity  Correlation across duration and cost uncertainties  Probabilistic branching  rarely attempted Discrete Risk

28 Aug 2012 PRT-148 Approved for Public Release15 of 35 5x5 Matrix Definitions* n Risk Management conventions* Consequence  1 Minimal or no impact  2 Additional resources < 5%  3 Additional resources = 5-7%  4 Additional resources = 7-10%  5 Additional resources > 10% Likelihood of Occurrence  1 Remote (10%)  2 Unlikely (30%)  3 Likely (50%)  4 Highly likely (70%)  5 Near certainty (90%) n Opportunities Should have a separate matrix to address potential opportunities to save (not addressed in our example) *Note: Taken from Risk Management Guide for U.S. DoD Acquisition Every Agency will set its own standards for these values Consequence

28 Aug 2012 PRT-148 Approved for Public Release16 of 35 The JACS Process: Assigning Uncertainties Risk Sched Cost Collect Sched Data Create Analysis Schedule Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Update Analysis Schedule Collect Cost Data Identify as TD or TI Map to Sched Activities Assess Duration Uncertainty Assess Cost Uncertainty Assess Event Cost and Duration Uncertainty

28 Aug 2012 PRT-148 Approved for Public Release17 of 35 n Most common method to address uncertainty is to assign distributions to uncertain elements and run a Monte Carlo simulation Objective Uncertainty Distributions  Derived from historical data  Something you can defend mathematically and historically Subjective Uncertainty Distributions  Based more on expert opinion than statistical analysis  Often necessary due to lack of information to characterize it objectively n Every duration, cost and consequence in the model is generally an estimate and therefore uncertain Time Independent Cost Uncertainty Time Dependent Cost (Burn Rate or Resource Utilization) Uncertainty Duration Uncertainty Discrete Risk Uncertainty –Probability of Occurrence n Uncertainty should be applied in a consistent manner across the entire model The Only Certainty is Uncertainty

28 Aug 2012 PRT-148 Approved for Public Release18 of 35 Typical Uncertainty Distributions DISTRIBUTIONTYPICAL APPLICATION KNOWLEDGE OF MOST LIKELY NUMBER OF PARAMETERS REQUIRED RECOMMENDED PARAMETERS Lognormal Default when no better info. Probability skewed right. Replicate another model result. Power OLS CER uncertainty. Mean or median known better than the mode (most likely) 2 50% (median) and high value (some tools have a 3 rd parameter : “Location”. By default, it is zero. Used to “slide” the lognormal left or right (even into negative region). Triangular Expert opinion. Finite min/max. Chance reduces towards endpoints. Skew possible. Labor rates, labor rate adjustments, factor methods Good idea 3 Low, mode, and high BetaPert Like triangular, but treats mode as 4 times more important than min or max. Very good idea 3 Low, mode, and high Beta Like triangular, but min/max region known better than mode. Not sure 4 Min, low, high, and max Normal Equal chance low/high. Unbounded in either direction Linear OLS CER uncertainty. Good idea, but unbounded in either direction 2 Mean/Median/Mode and high value Uniform Equal chance over uncertainty range. Finite min/max. No idea 2 Low and High (some tools require min and max) Note: Low/high are defined with an associated percentile (by default 15/85). Min/Max are the absolute lower/upper bound (also known as the 0/100). Some policies require truncation at zero.

28 Aug 2012 PRT-148 Approved for Public Release19 of 35 The Double Counting Time Dependent (TD) Cost Uncertainty Dilemma n Consider how TD cost is calculated Typically calculated using: Duration * Cost/Day (duration in days) Cost/Day can be derived from similar, completed project totals Cost/Day factor may already capture cost and duration uncertainty Uncertainty on Duration and Cost/Day factor may be double counting n However, basis for Cost/Day must be carefully understood Cost/Day factor may change as the duration changes  Shorter duration achieved by using more resources ($/day larger)  Shorter duration achieved by using more expensive resources ($/day larger)  Longer duration a consequence of scarce resources ($/day smaller) In these contexts, uncertainty on Duration and Cost/Day is appropriate  Correlation between the two should be considered as well

28 Aug 2012 PRT-148 Approved for Public Release20 of 35 The JACS Process: Apply Correlation, Validate then Run Risk Sched Cost Collect Sched Data Create Analysis Schedule Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Update Analysis Schedule Collect Cost Data Identify as TD or TI Map to Sched Activities Assess Duration Uncertainty Validate File Run Analysis Assess Cost Uncertainty Assess Event Cost and Duration Uncertainty Apply Correlation

28 Aug 2012 PRT-148 Approved for Public Release21 of 35 n JACS models developed in any tool will have limited “functional” correlation between uncertain elements (correlation due to model mathematics) n Consider applying correlation to ensure elements that should, do move together n Tools should allow application of correlation across any uncertain elements Just because you can apply correlation, does not mean you should! Correlating Dur with TD may be double counting if TD is modeled as a function of Duration n Ideally, measure the correlation present in the model first, then apply as needed Apply Correlation

28 Aug 2012 PRT-148 Approved for Public Release22 of 35 n Prior to running an integrated simulation, the user should review the file to ensure that there are no potential issues within the file n There are many commercial tools that will perform a schedule health assessment These tools look for violations of schedule model best practices ( e.g., task without predecessor) n A JACS Health check is more comprehensive and should also uncover issues such as: Critical issues: e.g., cost not phased, invalid uncertainty, duration with no cost, invalid correlation Warnings: e.g., uncertainty on zero cost, risk event with zero probability, baseline outside uncertainty Information: e.g., extraordinary float, risk event turned off, duration without uncertainty, no correlation Perform a Comprehensive Health Check to Ensure Model Validity

28 Aug 2012 PRT-148 Approved for Public Release23 of 35 The JACS Process Risk Sched Cost Collect Sched Data Create Analysis Schedule Collect Risk Data Assign Likelihood, Estimate Impact Map to Sched Activities Update Analysis Schedule Collect Cost Data Identify as TD or TI Map to Sched Activities Assess Duration Uncertainty Validate File Run Analysis Assess Cost Uncertainty Assess Event Cost and Duration Uncertainty Apply Correlation

28 Aug 2012 PRT-148 Approved for Public Release24 of 35 n What is JACS? n Overview of the JACS modeling process n Key reports from a well constructed JACS model n Concluding remarks Agenda

28 Aug 2012 PRT-148 Approved for Public Release25 of 35 Typical Output from a Cost Only Uncertainty Analysis n Cost uncertainty is generally performed on total costs n Rarely linked to schedule uncertainties Unable to relate a specific cost result to a specific schedule result n No insight into uncertainty by year, or the impact of schedule slips

28 Aug 2012 PRT-148 Approved for Public Release26 of 35 A JACS Model Relates Uncertain Cost with Uncertain Duration New View Shows Cost Aligned with Schedule 70% Cost Confidence Level (CCL) Indicates Reserves Capture Schedule Growth Lower left identifies joint probability of meeting BOTH 70% cost and schedule (58.4%) Each dot is the cost and schedule result from one trial

28 Aug 2012 PRT-148 Approved for Public Release27 of 35 Annual Funding Charts Point Estimate vs Annual Uncertainty n The JACS Model provides total and annual uncertainty results n Identifies both WHAT the uncertainty is and WHEN it is

28 Aug 2012 PRT-148 Approved for Public Release28 of 35 Additional Views Are Possible Point Estimate similar to Mean Need to use Reserves to move point estimate towards the Mean

28 Aug 2012 PRT-148 Approved for Public Release29 of 35 Additional Views Are Possible n There are many tools available and they all produce some version of most of these charts

28 Aug 2012 PRT-148 Approved for Public Release30 of 35 n The term “Tornado” is used in a variety of contexts across schedule, cost and general uncertainty analysis; tool documentation and the literature n Various definitions include: Find Cost Drivers: Create a low and high “what-if” case for every “cost driver” based on their 10/90% values (evaluated one at a time)…find driver that has most impact  Problem: ignores correlation effects; does not address drivers influenced by multiple distributions Find Uncertainty Drivers: Measure the correlation between each defined input distribution and the output of interest. The highest correlation identifies find distribution that has the most impact on the total uncertainty.  Problem: Element that is highly correlated with output may have nothing to do with that output Hybrid: Sort the trial results by cost driver (one at a time) to find associated bounds on the output mean or selected percentile  Problem: How you bin the trials has massive effect on results; may require huge number of trials to obtain stable results Brute Force: Use one of the above methods to find the top 10, then run the simulation 10 times turning one at a time and record the impact on the output of interest.  Problem: Very tedious, time consuming and may be misleading if its loss triggers unexpected conditions in the simulation n Conclusion: Beware of the Tornado chart! Most Misunderstood Chart: Tornado

28 Aug 2012 PRT-148 Approved for Public Release31 of 35 n Total Cost Drivers: (left chart) Measure impact on inflated results, not constant year cost results Run simulation to find combined uncertainty results for 10/90% n Total Uncertainty Drivers: (right chart) Adjust the algorithm to account for applied correlations Identify how many trials used (demonstrate sufficient) n Notice the answers are quite different Tornado Chart Best Practices 7k Trials Required

28 Aug 2012 PRT-148 Approved for Public Release32 of 35 Prototype Integrated Time-Based View of Costs, Schedule, Budget, and Risks n Solid line is the point estimate (BCWS) n Dots are cost/schedule uncertainty results at various milestones n X- Axis identifies when key risk register events occur Size of symbol indicates impact, color indicates probability (or type)

28 Aug 2012 PRT-148 Approved for Public Release33 of 35 n What is JACS? n Overview of the JACS modeling process n Key reports from a well constructed JACS model n Concluding remarks Agenda

28 Aug 2012 PRT-148 Approved for Public Release34 of 35 What Data Do We Need? Keep it simple and use what you have n Schedule + Risks + Costs = JCL Model n Can use any of the below, as long as you have one data source for each category n Schedule Detailed IMS Simple schedule with just a few moving parts n Costs – preferably time phased Budget data Lower level cost data (LCC databases) / EVM data Parametric costs n “Risks” Risk management system What –if’s Basic uncertainty

28 Aug 2012 PRT-148 Approved for Public Release35 of 35 Why JACS is Becoming Popular n Delivers an integrated view to Project Managers: Schedule probability of success Cost probability of success Impact of discrete program risks Results of any number of what-if scenarios Both total and annual funding reserve requirements n For NASA: Regulatory requirement ( E) Identify a cost and schedule range by milestone KDP (~milestone) B Baseline program to a specific joint probability level by KDP (~milestone) C

28 Aug 2012 PRT-148 Approved for Public Release36 of 35 Questions?Questions?