Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cost Drivers Learning Event, 2 nd November 2005 1 Correlation Tutorial Raymond Covert, MCR, LLC Timothy Anderson, The Aerospace Corporation.

Similar presentations


Presentation on theme: "Cost Drivers Learning Event, 2 nd November 2005 1 Correlation Tutorial Raymond Covert, MCR, LLC Timothy Anderson, The Aerospace Corporation."— Presentation transcript:

1 Cost Drivers Learning Event, 2 nd November Correlation Tutorial Raymond Covert, MCR, LLC Timothy Anderson, The Aerospace Corporation This tutorial was developed by the authors at: The Aerospace Corporation Conference Center Drive, Suite 600 Chantilly, VA Copyright © 2004 The Aerospace Corporation

2 Cost Drivers Learning Event, 2 nd November September Outline 1. Introduction to Correlation in Risk Analysis Introduction to Correlation in Risk Analysis 2. A Statistical View of Cost Analysis A Statistical View of Cost Analysis 3. Types of Correlation Types of Correlation 4. The Correlation Matrix The Correlation Matrix 5. Deriving Correlation Coefficients Deriving Correlation Coefficients

3 Cost Drivers Learning Event, 2 nd November Part 1 Introduction to Correlation in Risk Analysis Purpose of Section - To Answer These 6 Questions About Correlation: 1. Who Should Understand It 2. What Is It 3. Why Is It Used 4. Where Is It Used 5. When Is It Used 6. How Is It Used

4 Cost Drivers Learning Event, 2 nd November September Who and What Who Should Understand Correlation ? Correlation should be understood by all cost analysts performing quantitative cost risk analysis. What is Correlation? Ref. 1 A measure of association between two variables. It measures how strongly the variables are related, or change, with each other. If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated. The most common statistic for measuring association is the Pearson correlation coefficient,  P. 1)

5 Cost Drivers Learning Event, 2 nd November September Correlation in Risk Analysis (2) Why is correlation used? To quantify the effects of statistical dependence when performing algebra on random variables. It has a large impact on the statistical properties of the results, particularly when many random variables are involved. Example: Dice Roll. What happens when we roll 2 dice and add their result? Assume 3 cases:  Case 1: Uncorrelated. Outcome of 1 die is independent from the other.  Case 2: Negatively correlated. Outcome of 1 die relate to the outcome of the other. If one die is a “6”, the other must be “1”.  Case 3: Positively correlated. Outcome of 1 die is same as the other.

6 Cost Drivers Learning Event, 2 nd November September Example: Dice Roll Roll of the die gives an equal chance of getting an outcome (1,2,3,4,5 or 6) Equal, discrete probability Uniform discrete distribution of probabilities Variance,  2 = 3.5 What happens when we sum 2 correlated dice? Roll of Die Roll of Die Probability Probability of x, P(x) = 1/6

7 Cost Drivers Learning Event, 2 nd November September Example: Dice Roll Sum of Dice: Uncorrelated Sum of Dice Probability Sum of Dice: Correlation = Sum of Dice Probability Sum of Dice: Correlation = Sum of Dice Probability Case 1:  = 0 Case 2:  = -1 Case 2:  = +1 Triangular, discrete shape Moderate variance,  2 =6 Mean = 7 P(7) = 1 P(<>7)=0 No variance,  2 =0 Mean = 7 Uniform, discrete shape P(each even)=1/6, P(odd) =0 Wide variance,  2 =14 Mean = = = = 4

8 Cost Drivers Learning Event, 2 nd November September What We Learned From Dice Roll What we learned about the effects of correlation on sums of dice: It affects the variance and shape It doesn’t affect the mean  = 0 changes shape to a discrete triangular distribution  =-1 changes shape and removes variance  =+1 preserves shape, adds the most variance, and is the same as multiplying by 2 The sum of dice example used a discretely distributed random variable, but the same rules apply for continuously distributed random variables. Uniform Triangular Normal Lognormal Weibull

9 Cost Drivers Learning Event, 2 nd November September Where and When Correlation Is Used Where is correlation used? When performing algebra on random variables. Quantifying the effects of random variables in cost estimates.  Summing costs of WBS elements.  Multiplying costs by random variables (i.e. Inflation).  In exponentiation of one random variable with another (i.e. learning curves). When is correlation used? Whenever we have random variables in our estimates. When we use Monte Carlo Simulations. In analytic statistical sums.

10 Cost Drivers Learning Event, 2 nd November September How Correlation Is Used Directly Through a correlation matrix, Indirectly By neglecting correlation, we are defining  = 0 By “reusing” random variables, we are defining  = 1  Example: We define inflation as a random variable and use the same random variable throughout our cost estimate By multiplying random variables by a constant, we are defining  = 1  Example: We define spacecraft weight as a random variable and use fractions of it to define weights of different subsystems.

11 Cost Drivers Learning Event, 2 nd November September Part 1 Summary All cost analysts performing quantitative cost risk analysis should understand correlation Correlation measures how strongly the variables are related, or change, with each other. Correlation affects the variance and shape, but not the mean We use correlation frequently, but may not even know it

12 Cost Drivers Learning Event, 2 nd November Part 2 A Statistical View of Cost Analysis Purpose of Section: To Understand the following 1. Costs are uncertain quantities 2. Costs can be treated as random variables 3. How correlation affects variance

13 Cost Drivers Learning Event, 2 nd November September A Statistical View of Cost Analysis WBS Element Costs are Uncertain Quantities That Have “Probability” Distributions And Statistical Characteristics such as Mean, Median, Mode Costs are Random Variables Our Goal in Cost Risk Analysis is to Combine Element Cost Distributions to Generate Probability Distribution of Total Cost Use Monte-Carlo or Other Statistical Procedure Quantify Confidence in “Best” Estimate of Total Cost, e.g., Mode Read off Mean, 50th Percentile Cost, 70th Percentile Cost, etc., from Cumulative Distribution to Estimate Amount of Risk Dollars Needed

14 Cost Drivers Learning Event, 2 nd November September Elements of Risk Model Cost Drivers Risk Drivers Assumptions Cost Estimate Model Quantified Risk Our Cost / Risk Model Quantifies: Costs Effects of Uncertainty Uncertainty in Program Assumptions Risks to Program

15 Cost Drivers Learning Event, 2 nd November September Example Cost Drivers Component, Assembly, Propellant Weights Cooling Requirements Data-Processing Requirements Power Requirements Solar Array Area Orbit Altitude Thrust Requirements Special Mission Equipment

16 Cost Drivers Learning Event, 2 nd November September Example Risk Drivers Beyond-State-of-the-Art Technology Cooling Processing Survivability Power Laser Communications Unusual Production Requirements Large Quantities (Space Systems) Toxic Materials Yields Tight Schedules Undeveloped Technology Software Development Supplier Viability System Integration Multi-contractor Teams System Testing Limited Resources Program Funding Stretch- Out Premature Commitment to RDT&E Phase Unforeseen Events

17 Cost Drivers Learning Event, 2 nd November September Cost-element Probability Distributions Best Estimate Low Risk High Risk Low Cost, High Risk vs. High Cost, Low Risk Best Estimate Narrow Symmetric distribution: equal Probability of actual cost higher or lower than best estimate Wide, Right Skewed distribution Lower point estimate, but high probability of actual cost greater than point estimate These curves tell two very different stories Would you believe both could come from the same estimate? mode = mean mode

18 Cost Drivers Learning Event, 2 nd November September Correlation Affects the Variance are Costs of WBS Elements (Random Variables) and n = number of WBS elements Total Cost = Mean of Total Cost= Variance of Total Cost = = A Very Important Relationship

19 Cost Drivers Learning Event, 2 nd November September Variance Measures Dispersion Small Large XX Area = 1.00 XX

20 Cost Drivers Learning Event, 2 nd November September Does Correlation Matter? If WBS-Element Costs are Uncorrelated (all  ij = 0 ), Variance of Total Cost = If WBS-Element Costs are Correlated, Variance of Total Cost = – Positive Correlations Increase Dispersion – Negative Correlations Reduce Dispersion If (“Worst” Case) All Correlations, Variance of Total Cost = “Ignoring” Correlation Issue is Tantamount to Setting all  ij = 0

21 Cost Drivers Learning Event, 2 nd November September Yes, Correlation Matters Suppose for Simplicity There are n Cost Elements Each Total Cost

22 Cost Drivers Learning Event, 2 nd November September Magnitude of Correlation Impact Percent Underestimation of Total-Cost Sigma When Correlation Assumed to be 0 instead of  is 100% times... Percent Overestimation of Total-Cost Sigma When Correlation Assumed to be 1 instead of  is 100% times...

23 Cost Drivers Learning Event, 2 nd November September Maximum Possible Underestimation of Total-cost Sigma Percent Underestimated * 100% When Correlation Assumed to be 0 Instead of 

24 Cost Drivers Learning Event, 2 nd November September Maximum Possible Overestimation of Total-Cost Sigma Percent Overestimated When Correlation Assumed to be 1 Instead of 

25 Cost Drivers Learning Event, 2 nd November September Part 2 Summary In this section we learned: Costs are uncertain quantities Costs can be treated as random variables Correlation affects variance  Especially when we are summing large numbers of WBS elements Remember that in the total cost distribution: The means add The standard deviation is the square root of the variance The variance =

26 Cost Drivers Learning Event, 2 nd November Part 3 Types of Correlation Purpose of Section is to learn about: 1. Functional and Statistical Correlation 2. Pearson and Spearman Correlation

27 Cost Drivers Learning Event, 2 nd November September Types of Correlation Functional (Causal) Correlation Between cost drivers (Cost Engineering Tools) Between CERs (Cost dependent CERs – SEITPM) Statistical Correlation Between CER errors  Residual analysis (USCM,SSCM, NAFCOM)  Retro-ICE method  Estimated based on Number of WBS items Between Engineering drivers Between complexity, weight, power, etc.

28 Cost Drivers Learning Event, 2 nd November September Functional Correlation Between Cost Drivers Cost drivers are functionally correlated Beginning of Life Power, EPS Weight, Solar Array Weight, Battery Weight, RCS Weight, etc… Use Sizing equations from cost engineering tools to examine the causal relationship Two Good Examples: Electrical Power System Sizing Reaction Control System Sizing

29 Cost Drivers Learning Event, 2 nd November September Functional (Causal) Correlation Between Cost Drivers Example: Electrical Power System Sizing Equations in Design loop are functional correlations Power Requirement drives EPS Weight

30 Cost Drivers Learning Event, 2 nd November September Functional Correlation Between Cost Drivers Feedback Example: Reaction Control System Reaction Control Selection Natural Disturbance Torques SV Dimensions Solar Array Pointing Error Slew Rate Antenna Pointing Error Slew Rate Inertia: SV Body Antennas Solar Array Reaction Torque RCS Size RCS Power RCS Weight Gimbal Torques RCS Weight and size drives SV Body Inertia RCS Power drives Solar Array Inertia RCS Size drives SV Dimensions Equations in Design loop are functional correlations

31 Cost Drivers Learning Event, 2 nd November September There are two basic types of Cost Estimating Relationships (CERs). 1. Design Parameter Dependent: Subsystem Hardware (HW) CERs which use weight (or other design parameter) as a base. 2. Cost Dependent: Systems Engineering, Integration and Test, and Program Management (SEITPM) CERs which use estimated cost as a base. CERs are Used Serially in Risk Analysis. 1. Subsystem HW estimated costs are driven by subsystem estimated weights (or other cost drivers). 2. SEITPM estimated costs are driven by HW cost estimates. 3. So, the variance of the SEITPM cost estimate is functionally correlated to the HW cost estimates. Functional Correlation Between CERs

32 Cost Drivers Learning Event, 2 nd November September Mathematically Total estimate is a sum of the subsystem and SEITPM costs:* Subsystem estimates follow the form: SEITPM estimate follows the form: *Note: The “Total cost” is actually a sum of the subsystem and SEITPM costs. It is not the sum of all costs associated with a spacecraft. Error terms for SS estimates Error term for SEITPM Weight

33 Cost Drivers Learning Event, 2 nd November September Mathematically So, the SEITPM estimate is actually represented by: And the total estimate is represented by: Functional correlation SS CERsSEITPM

34 Cost Drivers Learning Event, 2 nd November September Statistical Correlation Between CER errors.  Residual analysis (USCM,SSCM, NAFCOM).  Retro-ICE method.  Estimated based on Number of WBS items. Between Engineering drivers. Between complexity, weight, power, etc. Look at 2 types of statistical correlation Pearson’s correlation Spearman’s correlation When you have data When you have to guess When you have data but don’t know functional relationships

35 Cost Drivers Learning Event, 2 nd November September Two Types of Statistical Correlation Pearson Product-Moment Linear Correlation if and only if X and Y are linearly related, i.e., the least- squares linear relationship between X and Y allows us to predict Y precisely, given X = proportion of variation in Y that can be explained on the basis of a least-squares linear relationship between X and Y if and only if the least-squares linear relationship between X and Y provides no ability to predict Y, given X Spearman Rank Correlation if and only if the largest value of X corresponds to the largest value of Y, the second largest,..., etc. if and only if the largest value of X corresponds to the smallest value of Y, etc. if and only if the rank of a particular X among all X values provides no ability to predict the rank of the corresponding Y among all Y values

36 Cost Drivers Learning Event, 2 nd November September Pearson “Product-Moment” Correlation Suppose X and Y are Two Random Variables are their Expected Values (“Means”) True Theorem: False Theorem: “Covariance” of X and Y “Variance” of X “Variance” of Y “Correlation” of X and Y =

37 Cost Drivers Learning Event, 2 nd November September Pearson Correlation Measures Linearity A Statistical Relationship Between Two Random Variables X and Y Realizations of Y, Given Actual Values of X : X Y 0 X Y 0

38 Cost Drivers Learning Event, 2 nd November September Spearman Rank Correlation Coefficient Data Structure Statistics Theorem: Spearman Rank Correlation Coefficient Equals Pearson (Linear) Correlation Coefficient Calculated Between the Two Sets of Ranks  s

39 Cost Drivers Learning Event, 2 nd November September Linear vs. Rank Correlation     s     s LINEARPOWER“KNEE” ROOT DECAY w/ OUTLIER RANDOM w/ OUTLIER More Nonlinear Linear Data gives similar  and  s

40 Cost Drivers Learning Event, 2 nd November September What Does Correlation Measure? PEARSON Correlation Measures Extent of LINEARITY of a Relationship Between Two Random Variables SPEARMAN Correlation Measures Extent of MONOTONICITY of a Relationship Between Two Random Variables A way to remember: …L M N O P Q R S… Rank Is Spearman Linear Is Pearson

41 Cost Drivers Learning Event, 2 nd November September Part 3 Summary Two types of correlation Functional and Statistical Correlation Functional correlation affects Cost Drivers and Cost Dependent CERs Statistical correlation is an observed relationship between data Two types of Correlation Statistics Pearson and Spearman Correlation or Linear and Rank Correlation They are similar when the data is linear They are different when data is not linear

42 Cost Drivers Learning Event, 2 nd November Part 4 The Correlation Matrix Purpose of Section is to learn : 1. Correlation in Risk Rollups 2. Anatomy of a Correlation Matrix 3. How to use a Correlation Matrix 4. Which Common Cost Models Handle Correlation

43 Cost Drivers Learning Event, 2 nd November September Cost-Risk Procedure Use analytic or Statistical sampling methods to arrive at a total cost distribution WBS-ELEMENT TRIANGULAR DISTRIBUTIONS MERGE WBS-ELEMENT COST DISTRIBUTIONS INTO TOTAL-COST DISTRIBUTION BEST ESTIMATE COST (MOST LIKELY) 70th PERCENTILE COST RISK DOLLAR S $ Note: Addition of risk dollars brings confidence that total appropriation (best estimate plus risk dollars) is sufficient to fund program. L1L1 B1B1 H1H1 $ L 2 = B 2 H2H2 $ L3L3 B3B3 H3H3 $

44 Cost Drivers Learning Event, 2 nd November September Total Cost Variance Remember from Part 1, the Total cost variance,   l l    T  n  Correlation matrix (full matrix) n  Vector of standard deviations (cost space) n Excel Commands u SIGMA_TOT=SQRT(MMULT(MMULT(TRANSPOSE(SIGMA),RHO),SIGMA)) Correlation is Essential in calculating variance!

45 Cost Drivers Learning Event, 2 nd November September Representing Correlation Matrices Full Matrix (Have to use this when you use analytic function: ) Upper Triangular: Lower Triangular: All 3 representations mean the same thing    T 

46 Cost Drivers Learning Event, 2 nd November September Representing Correlation Matrices Single value shorthand: This means all of the off diagonal terms are the same value The Rules: Always positive definite Diagonal terms always 1.0 Off diagonal terms are correlation values Columns and rows are transposed,  j,k =  k,j Now for some practical examples   

47 Cost Drivers Learning Event, 2 nd November September Cost Risk Rollup Procedure WBS Items Segment (Launch, Space, Ground) Want to form a Distribution for Total Cost of Program Spacecraft Bus Subsystems Payload Subsystems Payload SEITPM (function of payload subsystem cost) System (Spacecraft Bus, Payload) Subsystem 1 Subsystem 2 Subsystem N    f($) Subsystem 1 Subsystem 2 Subsystem N f($) Launch Other Elements Spacecraft Bus SEITPM (function of spacecraft bus subsystem cost) Inter-element Correlation needed for All Rollup (Summed) Costs

48 Cost Drivers Learning Event, 2 nd November September Spacecraft Bus: USCM7 Correlation Coefficients Correlation coefficients for USCM7 Weight based, Mean Unbiased Percentage Error (MUPE) CERs Average correlation coefficient = These correlation coefficients should not be used for all spacecraft cost models

49 Cost Drivers Learning Event, 2 nd November September The “Big” WBS Case Study (ISS Risk Estimate) Suppose a risk analyst diligently applies distributions to all costs at the “level of estimating” – this is good. Assume that: There are 300 cost elements (N=300) There are about four cost elements in each subsystem (n=4) There are (N/n = 75 subsystems) Correlation is defined between all elements within a subsystem This means: That WBS elements are correlated Only about 1% of the cost elements are correlated Risk is very narrow and understated The correlation appears “just-off-the diagonal” of the correlation matrix – This is bad.

50 Cost Drivers Learning Event, 2 nd November September “Just-Off-Diagonal” Correlation  Some tools cannot support this function  Some nominal statistical correlation does exist  Even a few percent makes a big difference with a big WBS These Inter-Subsystem WBS Elements are Effectively Uncorrelated These Intra-Subsystem WBS Elements are Correlated

51 Cost Drivers Learning Event, 2 nd November September How to Use Correlation Matrices Typically, we wouldn’t want to define all of the correlation coefficients for a big WBS (>10 elements) We can break it up into parts, get the statistics and then sum at higher levels Reduces the size of correlation matrices Provides Risk Breakout by WBS Summary Level Lets use an example of a “Big” WBS with 40 elements

52 Cost Drivers Learning Event, 2 nd November September Yuck 40 Individual WBS Elements and the correlation matrix Imagine 300 WBS Elements!

53 Cost Drivers Learning Event, 2 nd November September Big Correlation Matrix Layout AA BA CA DA EA AB BB CB DB EB AC BC CC DC EC AD BD CD DD ED AE BE CE DE EE A= SETPM (5 elements) B= Space Segment (20 elements) C= Ground Segment (5 elements) D= Launch Segment (5) elements) E= Operations Segment (5 elements) Each Block Represents a group of inter element correlations The full matrix requires (40*39)/2=780 different correlations

54 Cost Drivers Learning Event, 2 nd November September Multilevel Risk Look at the problem one small set of pieces at a time Now do the same for SPACE, LAUNCH, GROUND, O&M SEITPM Mean = Sum(Means) SEITPM Sigma =SQRT(MMULT(MMULT(TRANSPOSE(sigma),correl_matrix),sigma))  SEITPM

55 Cost Drivers Learning Event, 2 nd November September Space Element Risk In the Space Element, first break-out the Bus calculation Space Vehicle SEITPM is one line item, Let’s assume we already calculated mean and sigma for the Payload, like we did for the Bus  S/C Bus

56 Cost Drivers Learning Event, 2 nd November September Space Element Risk Now we roll-up 3 Items: Use a small correlation matrix: The result is: Cost PDF

57 Cost Drivers Learning Event, 2 nd November September Economics of Multi-Level Risk After summing all of the elements, we used : 144 Correlation Coefficients vs. 780 (one big matrix) Views of Risk at all roll-up levels Easier to obtain values for correlation coefficients We will discuss this in the next part

58 Cost Drivers Learning Event, 2 nd November September What We Just Did AA BA CA DA EA AB BB CB DB EB AC BC CC DC EC AD BD CD DD ED AE BE CE DE EE A= SETPM (5 elements) B= Space Segment (20 elements) C= Ground Segment (5 elements) D= Launch Segment (5) elements) E= Operations Segment (5 elements) Relied on AA, BB, CC, DD, and EE correlation

59 Cost Drivers Learning Event, 2 nd November September Mathematically Step1: Calculate ; Where, and Step 2: Need correlation coefficients of partition AA, BB and all  s to calculate

60 Cost Drivers Learning Event, 2 nd November September Mathematically Step 3: Calculate total variance using This is useful when : We know the correlation between subsystem elements But not the correlation between subsystems from different elements to each other (i.e., thermal control SS in spacecraft to ground Command and control CSCIs) But do have an idea of correlation between higher-level elements like space to ground.

61 Cost Drivers Learning Event, 2 nd November September Mathematically

62 Cost Drivers Learning Event, 2 nd November September Models That Handle Correlation PRICE* NAFCOM* SSCM* USCM + Others Cannot Full Partial Simulation AnalyticDiscrete and CB Monte Carlo Simulators Cost Model w/ Risk* SEER* RI$K SICM With Analytic or Simulation Method & correlation  Study Type of Risk Analysis Method Level of Correlation Specification © 2003 The Aerospace Corporation

63 Cost Drivers Learning Event, 2 nd November September Part 4 Summary Showed how correlation is used in Risk Rollups Provided the anatomy of a correlation matrix 1’s on diagonals Correlation coefficients on off-diagonals Rows and columns are transposes of each other How to use a correlation matrix Breaking down big risk jobs into smaller pieces Easier to understand Easier to correlate Showed which common cost models handle correlation

64 Cost Drivers Learning Event, 2 nd November Part 5 Deriving Correlation Coefficients Purpose of Section is to learn how to derive correlation coefficients: 1. When data is available 2. When you have to make an educated guess

65 Cost Drivers Learning Event, 2 nd November September Deriving Correlation Coefficients 2 Ways to derive correlation coefficients  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

66 Cost Drivers Learning Event, 2 nd November Determining Correlation When Data is Available  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

67 Cost Drivers Learning Event, 2 nd November September Determining Correlation When Data is Available Statistical Correlation Residual analysis (USCM,SSCM, NAFCOM) (HARD)  Need Database of cost and cost drivers+ CERs+ CER errors  Only have good bus correlations with standard models Retro-ICE method (HARD)  Need actual cost data from several similar programs + similar WBS structure+ total error+ similar models Estimated based on Number of WBS items (EASY)  Need number of WBS items + typical uncertainty  Strong function of number of correlated elements  Decreases with number of correlated elements Functional (Causal) Correlation Between cost drivers (HARD)  Need Cost Engineering Tools (CDC, SWAP model) Between CERs (EASY)  Use cost dependent CERs (SEITPM, etc) linked to summary costs in model © 2003 The Aerospace Corporation

68 Cost Drivers Learning Event, 2 nd November Residual Analysis Method  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

69 Cost Drivers Learning Event, 2 nd November September Statistical Correlation From Residual Analysis Percentage error or standard error are a measure of residual errors Uncertainty and risk calculations Use residual errors to represent uncertainty Correlation between residuals Cost vs. Weight Weight (lbs) Cost ($K)

70 Cost Drivers Learning Event, 2 nd November September Deriving Correlation Coefficients Sample calculation using randomly generated numbers Error X i and Error Y i represent regression residuals for 2 CERs (X and Y) for 8 programs

71 Cost Drivers Learning Event, 2 nd November Retro-ICE Method  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

72 Cost Drivers Learning Event, 2 nd November September Retro-ICE Method Another way to look at correlation is the effect on the variance of the total vs. the variance of the components Example: SPACE Segment Contains 3 elements: Space SEITPM, Spacecraft Bus, Payload What you will need: Actuals for some programs (8 or so) Estimates using your “new” cost model or method (You will be doing a Retro-ICE) Use the equation for Pearson correlation:

73 Cost Drivers Learning Event, 2 nd November September Retro-ICE Example Start with a table of actuals and estimates for 8 programs and the 3 WBS elements you wish to determine correlation 8 Programs Actual CostsRe-Estimated Costs (From Retro-ICEs)

74 Cost Drivers Learning Event, 2 nd November September Retro-ICE Example Calculate the Residuals (Actuals – Estimate) For Each WBS Element (and each program) For the Total Find the standard deviation, , of each set of residuals For Each WBS Element (8.05, 6.63, 21.26) For the Total (30.53)

75 Cost Drivers Learning Event, 2 nd November September Retro-ICE Example Construct the correlation matrix Ones on the diagonal Use Excel “CORREL” function on the off-diagonals Remember the row/column transpose rule Check your work: The result should match the standard deviation of the total, ( )^2

76 Cost Drivers Learning Event, 2 nd November Effective Correlation  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

77 Cost Drivers Learning Event, 2 nd November September Effective Correlation Effective correlation is different from average correlation, or the average of the correlation values in the upper (or lower) triangle of the correlation matrix Effective correlation is weighted by the value of the standard deviation of the constituent elements The effective correlation may be much different from the average correlation Look at SSCM as an example © 2003 The Aerospace Corporation

78 Cost Drivers Learning Event, 2 nd November September Effective Correlation SSCM has an average correlation of 0.04, but an effective correlation of 0.10 Effective correlation was calculated with SSCM the following way: 1. Calculate the total cost error of SSCM For each data point in the database:  Calculate the sum of actual cost database  Calculate the sum of the estimated costs  Determine the percentage error, and take the average 2 For each data point, calculate the SSCM error by multiplying the sum of actual costs by the SSCM percent error,  SSCM and square to get   TOT © 2003 The Aerospace Corporation

79 Cost Drivers Learning Event, 2 nd November September Then, for each data point, calculate  i by multiplying the actual cost for each WBS element by its respective percent error 4. Calculate the dot product of  i with itself to get   i 5. Now use the following formula to get the effective correlation,  eff for that data point. 6. Finally, get the average of the effective correlations to get  eff for the model The  eff for SSCM is 0.10 We should figure this out for all of our models! Effective Correlation © 2003 The Aerospace Corporation

80 Cost Drivers Learning Event, 2 nd November September Effective Correlation with the Retro ICE Method Remember the Retro Ice Example? Use the effective correlation equation to solve for  eff

81 Cost Drivers Learning Event, 2 nd November September Effective Correlation with the Retro ICE Method Retro ICE correlation Matrix Average  = Retro ICE effective correlation  eff =0.508

82 Cost Drivers Learning Event, 2 nd November Determining Correlation When Data is Not Available  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

83 Cost Drivers Learning Event, 2 nd November September The Problem It is not always possible to calculate statistical correlation between WBS elements. May be insufficient data to determine statistical correlation. May be no known functional relationship between WBS elements. Yet, there may be reason to believe increases or decreases in the cost of a certain WBS element are likely to cause corresponding increases or decreases in the cost of another WBS element. In cases such as these, it is still desirable to construct a correlation matrix in order to ensure a truer picture of the total cost variance.

84 Cost Drivers Learning Event, 2 nd November September Potential Solutions What can you do if you cannot construct a correlation matrix from statistical or empirical means? Assume independence  Same as a correlation matrix of zeros.  Extremely easy – but, as we have shown, it is WRONG! Use Knee-in-Curve Method (Steve Book’s Rule of Thumb)  When in doubt, assume all correlation values are 0.2.  Captures about 80% of the variance compared to assuming independence. Easy to do. Use the “N-effect”  Modulate our guess at correlation by preserving total error of the estimate Develop a subjective correlation matrix  Excellent results if you can do it accurately.  Can fill out an entire correlation matrix this way, but is somewhat difficult.

85 Cost Drivers Learning Event, 2 nd November Knee-in-Curve Method (Steve Book’s Rule of Thumb)  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

86 Cost Drivers Learning Event, 2 nd November September Steve Book’s Rule of Thumb According to Dr. Steve Book… From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA

87 Cost Drivers Learning Event, 2 nd November September Steve Book’s Rule of Thumb Dr. Book plotted the theoretical underestimation of percent total cost standard deviation when correlation is assumed to be zero rather than its true value, . From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA

88 Cost Drivers Learning Event, 2 nd November September Steve Book’s Rule of Thumb For example, given a 30 x 30 correlation matrix in which each actual correlation coefficient is 0.2, if you were to instead assume each correlation coefficient’s value is zero, then you would underestimate the standard deviation of the resulting cost probability distribution by about 60%. Dr. Book argues that since the “knee” of these curves lies at about 0.2, then it is better to populate an unknown correlation matrix with 0.2’s, or 0.3’s, rather than zeros. Doing so will reasonably capture a substantial amount of cost variance over doing nothing.

89 Cost Drivers Learning Event, 2 nd November September Steve Book’s Rule of Thumb Again, according to Dr. Book… From: 1999 Cost Risk Analysis Seminar, Manhattan Beach, CA

90 Cost Drivers Learning Event, 2 nd November “N-effect” Correlation  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

91 Cost Drivers Learning Event, 2 nd November September “N-effect” Correlation As N increases, the effective correlation (  eff ) will decrease in reaction to the central limit theorem. This is the “N-effect” Why? There is a fundamental limit to the predictive capability of our CERs. Just by breaking the WBS up into more pieces doesn’t improve our estimates.

92 Cost Drivers Learning Event, 2 nd November September “N-effect” Correlation Average Correlation* in models seem to be sensitive to number (N) of CERs As N increases,  decreases © 2003 The Aerospace Corporation Maximum Possible Underestimation of Total-Cost Sigma Actual Correlation Percent Underestimated NAFCOM N= 55 USCM7 Bus N= 19 USCM7 FU Bus N= 11 USCM8 Bus N= 17 USCM8 Comm N= 13 SSCM N= 9 The average correlation is different from the effective correlation, but the effect is similar

93 Cost Drivers Learning Event, 2 nd November September Determining Correlation from the Number of WBS Items There appears to be a trend between the number of WBS Elements (N) in a cost model and the derived average correlation coefficient (  AVG ) and effective correlation  EFF  EFF is a single number used to fill the correlation matrix As N increases,  EFF decreases We looked at the following models: NAFCOM (NASA/ Air Force Cost Model) USCM7 (Unmanned Space Vehicle Cost Model, Ver. 7) USCM8 (Unmanned Space Vehicle Cost Model, Ver. 8) SSCM (Small Satellite Cost Model)  

94 Cost Drivers Learning Event, 2 nd November September Determining Correlation from Number of WBS Items If we see a trend in the chart of percent under-estimation of sigma vs. effective correlation, we have a sound basis for determining correlations when the number of WBS elements grows. If the actual percent underestimated is k then the N-effect correlation  N for a model with N CERs would be: So, for k=50%: © 2003 The Aerospace Corporation

95 Cost Drivers Learning Event, 2 nd November Causal Guess Method of Subjective Correlation (Tim Anderson’s Method)  Data Available: (CADRE, CERs) No Data: Educated Guess Residual Analysis Retro- ICE Causal Guess N-Effect Guess StatisticalNon-Statistical Effective  Knee in curve (Steve Book Method)

96 Cost Drivers Learning Event, 2 nd November September Subjective Correlation It has previously been shown that it is possible to derive the empirical residual correlation coefficients of a cost model such as USCM, NAFCOM or SSCM. However, this method requires exclusive use of either of these two cost models to be effective. One alternative method is to subjectively develop approximate correlation coefficients between WBS elements. This can be as simple as determining whether any two WBS elements are correlated by a small amount, or by a large amount, and whether that correlation is positive or negative.

97 Cost Drivers Learning Event, 2 nd November September Subjective Correlation An example of a subjective correlation decision table might look like the following: For example, if you believe two WBS elements have a small amount of positive correlation, then you would choose a correlation value of 0.3. Positive correlation Negative correlation Uncorrelated 0 0 Small amount of correlation Large amount of correlation Subjective Correlation Coefficients

98 Cost Drivers Learning Event, 2 nd November September Subjective Correlation Using this technique, it is only necessary to make the following argument between any two WBS elements: Thus, if the answer were that a change in the cost of one WBS element might cause a minor, similar change in the other WBS element, then one would assign a correlation value of 0.3 between the two WBS elements. “If circumstances cause the cost of one WBS element to change, is the other WBS element likely to change also? If so, how substantially and in what direction?”

99 Cost Drivers Learning Event, 2 nd November September But Does It Work? The subjective scoring scheme shown previously is based on averages. As the figure below shows, the values 0.0, ±0.3, and ±0.75 are the average values in each of their respective ranges. l The idea is this: While one might not know the true correlation between WBS elements, if instead one can subjectively bucket the correlation into one of these five ranges, then a correlation matrix composed of these averages should give approximately the same results as if the true correlations were known.

100 Cost Drivers Learning Event, 2 nd November September But Does It Work? To test the hypothesis, suppose we substitute the true correlation coefficients in a typical correlation matrix with their corresponding averages from the subjective correlation table. If the resulting cost probability distributions are similar, then we can use the technique with some confidence. An important point, however, is that the subjectively derived values should reflect the actual values. If this is true, then this method should give a more precise answer than Steve Book’s Rule of Thumb method. We will see in the following example

101 Cost Drivers Learning Event, 2 nd November Application of Correlation Methods When Data is Not Available Cost Risk Analysis of A Small Earth Orbiting Visible Imaging Sensor

102 Cost Drivers Learning Event, 2 nd November September Example Consider an estimate developed using the following cost model: Reprinted from Space Mission Analysis And Design, 3 rd Edition. Wertz and Larson

103 Cost Drivers Learning Event, 2 nd November September Input Variables Suppose this spacecraft has the following set of (mean) input variables:

104 Cost Drivers Learning Event, 2 nd November September Functional Relationships Integration and Test, Program Management and LOOS are functionally correlated to PL and S/C bus cost The input variables are functionally correlated through the following sizing relationships with their error terms (  ) Payload Wt. = 200 * (Aperture Diameter)^1.5 +  PL Structure Wt. = 0.4 * Payload Wt. +  STR Thermal Wt. = 0.05 * Payload Wt. +  TH BOLP = 5.0 * Payload Wt. +  BOLP EPS Wt. = 0.3 * BOLP +  EPS TTC Wt. = * Payload Wt. +  TTC ADCS Wt = 0.7 * Payload Wt. +  ADCS

105 Cost Drivers Learning Event, 2 nd November September Input Parameter Error Terms Aperture Diameter has a discrete 20% probability of being 1.0m, and a 80% probability of being 1.2m. Assume the error terms (  ) for the sizing relationships are all triangular probability density functions defined by Low = 0.9 Most Likely = 1.0 High=

106 Cost Drivers Learning Event, 2 nd November September Cost Estimate The resulting cost estimate has the following (deterministic) value:

107 Cost Drivers Learning Event, 2 nd November September “Actual” Correlation Matrix l Suppose the “actual” correlation matrix is as follows:

108 Cost Drivers Learning Event, 2 nd November September Total Cost Distribution: Actual Correlation Values Total cost distribution with “actual” correlation values: $M Lognormal: Mean = $534.7M Std Dev = $126.6M Lognormal: Mean = $534.7M Std Dev = $126.6M Note: This is the distribution that follows as a result of using the “actual” correlation coefficients. The standard deviation is 24% of the mean. Note: This is the distribution that follows as a result of using the “actual” correlation coefficients. The standard deviation is 24% of the mean. Frequency Chart ,000 Trials 0 Outliers Forecast: Total RDT&E + T1

109 Cost Drivers Learning Event, 2 nd November September Total Cost Distribution: No Statistical Correlation We have included functional correlation, but no statistical correlation between the error terms,  Under these circumstances, the total cost distribution has the following appearance and statistics: Frequency Chart ,000 Trials 0 Outliers Forecast: Total RDT&E + T1 Note: This is a narrow distribution. The standard deviation is 15% of the mean. This is very different from the percent errors of our CERs, which ranged from 34% to 61%. Note: This is a narrow distribution. The standard deviation is 15% of the mean. This is very different from the percent errors of our CERs, which ranged from 34% to 61%. $M Lognormal: Mean = $525.8M Std Dev = $79.7M Lognormal: Mean = $525.8M Std Dev = $79.7M

110 Cost Drivers Learning Event, 2 nd November September Total Cost Distribution: Knee-in-Curve Method Using Steve Book’s Rule of Thumb, the correlation is set to 0.2, and the total cost distribution has the following appearance and statistics: $M Note: This has caused the standard deviation to shift substantially compared to the zero correlation case. The standard deviation grew by over 76% from the uncorrelated case. Note: This has caused the standard deviation to shift substantially compared to the zero correlation case. The standard deviation grew by over 76% from the uncorrelated case. Frequency Chart ,000 Trials 1 Outlier Forecast: Total RDT&E + T1 Lognormal: Mean = $535.3M Std Dev = $148.2M Lognormal: Mean = $535.3M Std Dev = $148.2M

111 Cost Drivers Learning Event, 2 nd November September Total Cost Distribution: “N-Effect” Method Using the N-Effect method, the correlation is 0.125, and the total cost distribution has the following appearance and statistics: Note: This has caused the standard deviation to shift substantially compared to the zero correlation case. The standard deviation grew by over 59% from the uncorrelated case. This is almost exactly the Standard error of the “actual correlation” case. Note: This has caused the standard deviation to shift substantially compared to the zero correlation case. The standard deviation grew by over 59% from the uncorrelated case. This is almost exactly the Standard error of the “actual correlation” case. $M Frequency Chart ,000 Trials 0 Outliers Forecast: Total RDT&E + T1 Lognormal: Mean = $532.6M Std Dev = $127.0M Lognormal: Mean = $532.6M Std Dev = $127.0M

112 Cost Drivers Learning Event, 2 nd November September “Subjective” Correlation Matrix l The corresponding “subjective” correlation matrix using the causal guess method is as follows:

113 Cost Drivers Learning Event, 2 nd November September Total Cost Distribution: Causal Guess Subjective Values Total cost distribution with “subjective” correlation values: $M Note: This is the distribution that follows as a result of using the “subjective” correlation coefficients. The mean is nearly identical to the “actual” case, and the standard deviation has increased by approximately 4%. Note: This is the distribution that follows as a result of using the “subjective” correlation coefficients. The mean is nearly identical to the “actual” case, and the standard deviation has increased by approximately 4%. Lognormal: Mean = $531.2M Std Dev = $121.2M Lognormal: Mean = $531.2M Std Dev = $121.2M

114 Cost Drivers Learning Event, 2 nd November September Summary of Results meansigma Actual Uncorrelated Knee-in-curve N-Effect Causal guess

115 Cost Drivers Learning Event, 2 nd November September Summary of Results Ignoring correlation understated the total cost sigma The N-Effect and Causal Guess methods produced the best results The Knee-in curve method was close, but provided the largest variance

116 Cost Drivers Learning Event, 2 nd November September Part 5 Review We learned how to derive correlation coefficients: When data is available When you have to make an educated guess The Methods: Statistical (Data is available)  Residual analysis  Retro-ICE  Effective correlation,  eff When data is not available  Knee-in-curve method (Steve Book’s Rule of Thumb)  N-effect guess  Subjective guess (Tim Anderson’s Method) Example cost risk analysis using subjective methods

117 Cost Drivers Learning Event, 2 nd November End 


Download ppt "Cost Drivers Learning Event, 2 nd November 2005 1 Correlation Tutorial Raymond Covert, MCR, LLC Timothy Anderson, The Aerospace Corporation."

Similar presentations


Ads by Google