1. Pitfalls of Project Estimating:
An Applied Lesson from Dr. Deming’s Funnel Experiment
John Miller, PMP, CLSSBB
4/28/10

2. Dr. Deming’s Funnel Experiment What is a Process? Statistical Measures
Contents
Introduction
Dr. Deming’s Funnel Experiment
What is a Process?
Statistical Measures
Estimating Methods
Concepts
Averages and Standard Deviation
Data Points to Data Trends
Control Charts
Lessons from the Funnel Experiment
Using Excel to Avoid Common Mistakes
J. Miller, 4/28/10

3. Dr. Deming’s Funnel Experiment
Introduction
Parametric and evidenced-based estimating tools have become very sophisticated.
Many essential benefits of these tools can inexpensively be provided using Excel, but
project managers need to be aware of risks associated with improperly adjusting
estimating metrics.
Dr. Deming’s Funnel experiment serve’s as a model for improperly adjusting
estimating metrics. A simple example will demonstrate how statistically valid estimating methods can be duplicated in Excel.
Dr. Deming’s Funnel Experiment
“A experiment that demonstrates the effects of tampering [with a process]. Marbles are dropped through a funnel in an attempt to hit a flat-surfaced target below. The experiment shows that adjusting a stable process to compensate for an unstable result or an extraordinarily good result will produce output that is worse than if the process had been left alone.”
Donna C. Summers, Quality Management 2nd Edition, p 546
J. Miller, 4/28/10

4. DR. Deming’s Funnel Experiment
Funnel Comparison to Project Estimating
Funnel = Estimating process
Target = Goal -- project time goal or cost goal or resource goal
Marble = Actual project or estimate that creates measurable output – cost, schedule, etc.
Each marble drop = actual project or part of project
Moving the Funnel = Adjusting the project resources after each project to try to meet the target on the next project. (Ex. Adding people or tools, if schedules are being missed, removing resources if budgets are being missed)
Moving the Target = Adjusting the time or cost estimate estimating process after each project based on the results from the last project.
J. Miller, 4/28/10

5. DR. Deming’s Funnel Experiment
Funnel Comparison to Project Estimating   
Each of the four rules is discussed below.
Rule 1: Leave the funnel fixed over the target. Why don't we simply adjust the funnel after each drop so the next drop will be closer to the target? Don’t change the estimating process, accept the variation, the difference between the estimate and actual projects.
Rule 2: For every drop, the marble will come to rest a distance "z" from the target. Rule 2 is to move the funnel a distance -z from its last position. Move the funnel based on the funnels last position. Chang the estimating process (metrics) after each project or difference between estimate and actual.
Rule 3: Move the funnel a distance -z from the target after each drop of the marble that ends up a distance z from the target. Note that Rule 2 moves the funnel based on the funnel's last position. Rule 3 moves the funnel a distance from the target.
Rule 4: Rule 4 is simply to set the funnel over where the last drop came to rest.
J. Miller, 4/28/10

6. What is a Process?
A series of steps or actions that convert input to output.
Project estimating is a process.
Processes are described by statistical measures – numbers that describe “groups”.
Groups are made of data points.
What do we need to better understand this group of data points? Centering – Mean Spread - Stand Deviation Shape – Normality, data distribution (assume all distributions here are Normal)
We need data.
J. Miller, 4/28/10

7. Statistical Measures Dimensional attributes
Product or Process metrics such as:
# or % defects
% on time deliveries
Efficiency (productivity, ratios)
# units of output / unit of time (may also include inputs)
Time / # units of output
xx units / hr
Turn Around Time (TAT), xx Points / unit
Cost/Unit
Which can be averages?
J. Miller, 4/28/10

8. Estimating Methods PMBOK 3rd
6.4.2 Activity Duration Estimating: Tools and Techniques
.1 Expert Judgment
Activity durations are often difficult to estimate because of the number of factors that can influence them, such as resource levels or resource productivity. Expert judgment, guided by historical information, can be used whenever possible. The individual project team members may also provide duration estimate information or recommended maximum activity durations from prior similar projects. If such expertise is not available, the duration estimates are more uncertain and risky.
.2 Analogous Estimating
Analogous duration estimating means using the actual duration of a previous, similar schedule activity as the basis for estimating the duration of a future schedule activity. It is frequently used to estimate project duration when there is a limited amount of detailed information about the project for example, in the early phases of a project. Analogous estimating uses historical information
(Section ) and expert judgment. Analogous duration estimating is most reliable when the previous activities are similar in fact and not just in appearance, and the project team members preparing the estimates have the needed expertise.
J. Miller, 4/28/10

9. Estimating Methods PMBOK 3rd
6.4.2 Activity Duration Estimating: Tools and Techniques
.3 Parametric Estimating
Estimating the basis for activity durations can be quantitatively determined by multiplying the quantity of work to be performed by the productivity rate. For example, productivity rates can be estimated on a design project by the number of drawings times labor hours per drawing, or a cable installation in meters of cable times labor hours per meter. The total resource quantities are multiplied by the labor hours per work period or the production capability per work period, and divided by the number of those resources being applied to determine activity duration in work periods.
.4 Three-Point Estimates
The accuracy of the activity duration estimate can be improved by considering the amount of risk in the original estimate. Three-point estimates are based on determining three types of estimates:
• Most likely. The duration of the schedule activity, given the resources likely to be assigned, their productivity, realistic expectations of availability for the schedule activity, dependencies on other participants, and interruptions.
• Optimistic. The activity duration is based on a best-case scenario of what is described in the most likely estimate.
• Pessimistic. The activity duration is based on a worst-case scenario of what is described in the most likely estimate.
An activity duration estimate can be constructed by using an average of the three estimated durations. That average will often provide a more accurate activity duration estimate than the single point, most-likely estimate.
J. Miller, 4/28/10

10. Estimating Methods .1 Expert Judgment
Expert judgment, guided by historical information, can be used whenever possible.
.2 Analogous Estimating
Analogous estimating uses historical information and expert judgment.
.3 Parametric Estimating
Estimating the basis for activity durations can be quantitatively determined by multiplying the quantity of work to be performed by the productivity rate.
.4 Three-Point Estimates
An activity duration estimate can be constructed by using an average of the three estimated durations.
Mean The mean is the average data point value within a data set. To calculate the mean, add all of the individual data points then divide that figure by the total number of data points.
Published estimating methods overlook critical statistical attributes. What’s missing?
J. Miller, 4/28/10

11. Estimating Methods – Averages
What’s missing?
Mean
Standard Deviation
If I use the “average” of past projects to estimate future projects, what is virtually guaranteed?
J. Miller, 4/28/10

12. Concepts
Statistical measure – numbers that describe groups. A numerical value, such as standard deviation or average, that characterizes the sample or population from which it was derived.
What do we need to better understand this group of data points?
Centering – Mean
Spread - Stand Deviation
Shape – Normality, data distribution (assume all distributions here are Normal)
J. Miller, 4/28/10

13. Averages and Standard Deviation
Mean The mean is the average data point value within a data set. To calculate the mean, add all of the individual data points then divide that figure by the total number of data points.
Standard Deviation A statistic used to measure the variation in a distribution. Standard deviation is a measure of the spread of data in relation to the mean. It is the most common measure of the variability of a set of data.
J. Miller, 4/28/10

14. Averages and Standard Deviation
Standard Deviation Example
Suppose we wished to find the standard deviation of the data set consisting of the values 3, 7, 7, and 19.
Step 1: find the arithmetic mean (average) of 3, 7, 7, and 19 ( ) / 4 = 9.
Step 2: find the deviation of each number from the mean,
3 − 9 = − 6
7 − 9 = − 2
19 − 9 = 10.
Step 3: square each of the deviations, which amplifies large deviations and makes negative values positive,
( − 6)2 = 36
( − 2)2 = 4
(10)2 = 100.
Step 4: find the mean of those squared deviations,
( ) / 4 = 36.
Step 5: take the non-negative square root of the quotient (converting squared units back to regular units), so, the standard deviation of the set is 6.
J. Miller, 4/28/10

15. Standard Deviation Discovered
Assume: Mean = 30, Std Dev = 6
- 1 s = - 6
- 2 s = 2(-6) = -12
- 3 s = 3(-6) = -18
- 4 s = 4(-6) = -24
- 5 s = 5(-6) = -30
- 6 s = 6(-6) = -36 s -6 -5 -4 -3 -2 -1 30
1 s = 6
2 s = 2(6) = 12
3 s = 3(6) = 18
4 s = 4(6) = 24
5 s = 5(6) = 30
6 s = 6(6) = 36 s 1 2 3 4 5 6 +6
+12
+18
+24
+30
+36
-36
-30
-18
-12 -6
-24 36 42 48 54 60 66 -6 6 12 18 24
J. Miller, 4/28/10

16. Standard Deviation Discovered
Assume: Mean = 30, Std Dev = 6 30 36 42 48 54 60 66 -6 6 12 18 24
J. Miller, 4/28/10

17. Standard Deviation is Critical
Assume:
There are two projects, each using the Three Point Estimate Method
Project A Project B
Optimistic = Optimistic = 49
Most likely = Most likely = 50
Pessimistic = Pessimistic = 51 50 60 70 80 90
100 10 20 30 40 51 49
Explain the definition
Ask the class, ‘What is this symbol?’, referring to sigma. This is a lead in for the next slide. 99 1
Project A
Data points are 1, 50 ,99
Mean = 50
StdDev = 49
Project B
Data points are 49, 50, 51
Mean = 50
StdDev = 1
J. Miller, 4/28/10

18. Normal Distribution with Std Dev

19. Estimating Methods PMBOK 3rd
6.4.3 Activity Duration Estimating: Outputs
.1 Activity Duration Estimates
Activity duration estimates are quantitative assessments of the likely number of work periods that will be required to complete a schedule activity. Activity duration estimates include some indication of the range of possible results. For example:
• 2 weeks ± 2 Points to indicate that the schedule activity will take at least eight Points and no more than twelve (assuming a five-Point workweek).
• 15 percent probability of exceeding three weeks to indicate a high probability—85 percent—that the schedule activity will take three weeks or less.
Let’s take a closer look at making a valid estimate…
J. Miller, 4/28/10

20. Data Points to Data Trends
J. Miller, 4/28/10

21. Data Points to Data Trends
J. Miller, 4/28/10

22. Data Points to Data Trends
J. Miller, 4/28/10

23. Data Points to Data Trends
J. Miller, 4/28/10

24. Data Points to Data Trends
J. Miller, 4/28/10

25. Data Points to Data Trends
J. Miller, 4/28/10

26. Data Points to Data Trends
J. Miller, 4/28/10

27. Data Points to Data Trends
J. Miller, 4/28/10

28. Data Points to Data Trends
J. Miller, 4/28/10

29. Data Points to Data Trends
J. Miller, 4/28/10

30. Data Points to Data Trends
J. Miller, 4/28/10

31.  ? Data Points to Data Trends X UCL LCL P1 P2 P3 P4 P5 P6 P7
J. Miller, 4/28/10

32. Data Points to Data Trends
J. Miller, 4/28/10

33. Process Exercise
What about when we don’t have data for all process steps?
What if we had a way to estimate the data?

34. Creating a Control Chart
Define the metric: time/unit of output, delta of actual vs. planned time or cost, etc
Must be homogeneous
Data collection planning
Who will collect data?
What aspect of the process will be measured?
Where, or at what point in the process will the measurement be taken?
When or how frequently will the data be collected?
Why is this sample being taken?
How will the data be collected?
How many samples will be taken?
Collect the data
What if data isn’t collected for every project or period?

35. Creating a Control ChartOR
As long as the units are the same, the model works!

36. Creating a Control Chart
J. Miller, 4/28/10

37. Creating a Control Chart
Calculate the centerline – Sum of Means
Calculate the Upper and Lower Control Limits.
UCL = Sum of Means + 3 σ
Sum of Means
541.4
Sum of Variances
 503.3
Sq Rt of Variances
 22.4
LCL = Sum of Means - 3 σ
σ = Sq Rt of Sum of Variances
- 3 StdDev
- 2 StdDev
- 1 StdDev
Mean
+ 1 StdDev
+ 2 StdDev
+ 3 StdDev
-3* 22.4
-2* 22.4
-1* 22.4
1* 22.4
2* 22.4
3* 22.4
-67.2
-44.8
-22.4
541.4
22.4
44.8
67.2
474.2
496.6
519.0
563.8
586.2
608.6

38. Creating a Control Chart
- 3 StdDev
- 2 StdDev
- 1 StdDev
Mean
+ 1 StdDev
+ 2 StdDev
+ 3 StdDev
-3* 22.4
-2* 22.4
-1* 22.4
1* 22.4
2* 22.4
3* 22.4
-67.2
-44.8
-22.4
541.4
22.4
44.8
67.2
474.2
496.6
519.0
563.8
586.2
608.6
After calculating the control limits, place the center line and control limits on the chart
541.4
608.6
474.2

39. Creating a Control Chart
A Guide to the Project Management Body of Knowledge (PMBOK® Guide) Third Edition
2004 Project Management Institute, Four Campus Boulevard, Newtown Square, PA USA
J. Miller, 4/28/10

40. Determine if the Process is Stable Over Time
Check the date for trends
Check the date for patterns
J. Miller, 4/28/10

41. Process Stable Over Time?
Learn how to decide if the process is stable and if the data can be used for further analysis.
Data is analyzed using a control chart.
Point beyond UCL and LCL (beyond 3 sigma)
J. Miller, 4/28/10

42. Process Stable Over Time?
(2) Seven consecutive points on the same side of center line.
J. Miller, 4/28/10

43. Process Stable Over Time?
(3) Trend: 6 consecutive points steadily increasing or decreasing:
J. Miller, 4/28/10

44. Process Stable Over Time?
(4) Repeating pattern and cycles.
J. Miller, 4/28/10

45. Process Stable Over Time?
Can the above data be used for any further analysis and decision making?
Observations:
No point is beyond UCL and LCL
There is no pattern repetition
There is no continuous increase or decrease of 6 data points
There is NO 7 consecutive points on the same side of center line
So, this data can be used for future predictions
J. Miller, 4/28/10

46. Process Stable Over Time?
Can the above data be used for any further analysis and decision making
Observation :
There are points beyond UCL and LCL
There is no pattern repetition
There is no continuous increase or decrease of 6 data points
There is 7 consecutive points on the same side of center line
This tells, there is a special cause influencer, which is causing this to happen. Before making any predictions for the future, need to analyze the special cause
J. Miller, 4/28/10

47. Process Stable Over Time?
Can the above data be used for any further analysis and decision making :
Observation :
No point is beyond UCL and LCL
There is no pattern repetition
There is no continuous increase or decrease of 6 data points
There is 7 consecutive points on the same side of center line
So, Before using the data for future predictions, need to analyze the cause for data points to be on same side (during the period) and if required separate those points from the rest, in making the decision
J. Miller, 4/28/10

48. Determine if the Process is Stable Over Time
Check the date for trends
Check the date for patterns
Apply Lessons from the Funnel Experiment 
Rule 1: Leave the funnel fixed over the target. Why don't we simply adjust the funnel after each drop so the next drop will be closer to the target? Don’t change the estimating process, accept the variation, the difference between the estimate and actual projects.
Rule 2: For every drop, the marble will come to rest a distance "z" from the target. Rule 2 is to move the funnel a distance -z from its last position. Move the funnel based on the funnels last position. Chang the estimating process (metrics) after each project or difference between estimate and actual.
Rule 3: Move the funnel a distance -z from the target after each drop of the marble that ends up a distance z from the target. Note that Rule 2 moves the funnel based on the funnel's last position. Rule 3 moves the funnel a distance from the target.
Rule 4: Rule 4 is simply to set the funnel over where the last drop came to rest.
J. Miller, 4/28/10

49. Contact Questions? Using Excel to Avoid Common Mistakes John H. Miller
J. Miller, 4/28/10