1. A Step-Wise Approach to Elicit Triangular Distributions
SCEA Luncheon Series, Washington Area Chapter of SCEA
April 17, • Arlington, Virginia
A Step-Wise Approach to Elicit Triangular Distributions
Presented by:
Marc Greenberg
Office of Program Accountability and Risk Management (PARM)
Management Directorate, Department of Homeland Security (DHS)
The work in this paper is based upon:
Discussions I’ve had with many cost geeks over the years on how best to model the uncertainty of cost, duration, schedule, requirements, system characteristics, etc., when little or no data is available
My review of a wide variety of literature on expert judgment elicitation and probability distributions
Recent yet VERY preliminary work I’ve completed on modeling the elicitation process in a standardized, repeatable way (in Excel) to produce a triangular and a beta-PERT distributions

2. Risk, Uncertainty & Estimating
“It is better to be approximately right rather than precisely wrong.
Warren Buffett
Said to be a modern version of what was penned by Carveth Read in 1898 (Ref: Logic, Deductive and Inductive, 1898):
“It is better to be vaguely right than exactly wrong.”

3. Outline Purpose of Presentation Background
The Uncertainty Spectrum
Expert Judgment Elicitation (EE)
Continuous Distributions
More details on Triangular, Beta & Beta-PERT Distributions
Five Expert Elicitation (EE) Phases
Example: Estimate Morning Commute Time
Expert Elicitation (EE) to create a Triangular Distribution
With emphasis on Phase 4’s Q&A with Expert (2 iterations)
Convert Triangular Distribution into a Beta-PERT
Conclusion & Potential Improvements

4. Purpose of Presentation
Adapt / combine known methods to demonstrate an expert judgment elicitation process that …
Models expert’s inputs as a triangular distribution
12 questions to elicit required parameters for a bounded distribution
Not too complex to be impractical; not too simple to be too subjective
Incorporates techniques to account for expert bias
A repeatable Q&A process that is iterative & includes visual aids
Convert Triangular to Beta-PERT (if overconfidence was addressed)
Is structured in a way to help justify expert’s inputs
Expert must provide rationale for each of his/her responses
Using Risk Breakdown Structure, expert specifies each risk factor’s relative contribution to a given uncertainty (of cost, duration, reqt, etc.)
This paper will show one way of “extracting” expert opinion for estimating purposes. Nevertheless, as with most subjective methods, there are many ways to do this.

5. The Uncertainty Spectrum
Total Certainty = Complete information All known
Specific Uncertainty
Partial information Known unknowns
General Uncertainty
Total Uncertainty = No information Unknown unknowns
No Estimate Required
No Estimate Possible
Expert Opinion
Objective Probabilities
Subjective Probabilities
Data / Knowledge
Reference: Project Management Consulting by AEW Services, 2001
Expert opinion is useful when little information is available for system requirements, system characteristics, durations & cost

6. Expert Judgment Elicitation (EE)
Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen

7. Triangular Distribution
Used in situations were there is little or no data
Just requires the lowest (L), highest (H) and most likely values (M)
Each x-value has a respective f(x), sometimes called “Intensity” that forms the following PDF:
L, M & H are all that’s needed to calculate the Mean and Standard Deviation:
L M H

8. Beta Distribution
Bounded on [0,1] interval, scale to any interval & very flexible shape
b > a > 1, distribution is right skewed
Calculated Gamma values using Excel’s GAMMALN function:
Most schedule or cost estimates follow right skewed pattern. But how do we know a and b? Answer: Beta-PERT Distribution.
Sources: 1. Dr. Paul Garvey, Probability Methods for Cost Uncertainty Analysis, 2000
2. LaserLight Networks, Inc, “Beta Modeled PERT Schedules”

9. Beta-PERT Distribution
Requires lowest (L), highest (H) & most likely values (M)
Use L, M and H to calculate mean(m) and standard deviation (s) :
Use L, H, m and s To calculate shape parameters, a & b :
a and b are needed to define the Beta Function and compute the Beta Probability Density:
Beta Probability Density Function
(as shown in slide 9):
Typically for beta-PERT, lambda = 4
Deriving alpha & beta for Beta-PERT: Use the matching moment equations in which the minimum and maximum values are known to calculate shape factors which are consistent with the mean and standard deviation, one can derive alpha and beta.
Calculated Gamma values using Excel’s GAMMALN function:
Sources: 1. Dr. Paul Garvey, Probability Methods for Cost Uncertainty Analysis, 2000
2. LaserLight Networks, Inc, “Beta Modeled PERT Schedules”

10. Expert Elicitation (EE) Phases
Expert Elicitation consists of five phases: (note that Phases 4 & 5 are iterative)
Motivating the expert
Training (conditioning) the expert
Structuring objective, assumptions & process
Assessing (encoding) expert’s responses
Q&A – Expert’s technical opinion is elicited
Quantitative results w/ documented rationale
Verifying encoded values & documentation
Motivating phase. Ironically, helps eliminate motivational bias. The purpose of the first stage is to explain why and how the judgmental probabilities will be elicited, and how the results will be used in risk assessment. At this point the expert team will be informed of the items that will require quantification during the process. It is critical at this stage to ensure that all panel members can "play by the rules".
Training phase. Helps reduce unintentional bias. The purpose of the second phase is to encourage the Experts to think critically. The Facilitator will expose the experts to a short training session on concepts, objectives, and methods of eliciting judgmental probability. The Experts need to be cautioned to avoid the common biases and sources of error that people make when trying to quantify probability. The training phase will help the Experts to calibrate their responses by the use of common examples and examples related to their specific expertise.
Structuring phase. The Facilitator will work with the panel to review the technical issues to be addressed in the elicitation, refine the issues during discussions, and find an appropriate level of detail. Since people tend to be better at working with decomposed problems rather than systems, the Facilitator will work with the panel to segment the problem into manageable pieces. Additionally the Facilitator will work with the expert panel to decide on the specific parameters for which probabilities need to be assessed.
Assessing phase. During this phase each Expert's technical opinion is elicited. The Facilitator should direct the panel through an active approach to elicitation. This is achieved by having each expert associate numbers with descriptive statements, while avoiding intuitive or direct assignment. (See Table 1.)
Documentation. To stand the test of defensibility, the results of an expert elicitation should be reproducible, accountable, subject to peer review, neutral, and fair. Expert elicitation should be documented so that it is possible, in principal, to reproduce all the calculations involved the process. This means, calculation models should be fully reported and fully specified, questions asked of the experts should be reported, and responses of the experts should be tabulated. The source of all data and estimates in the study should be traceable to a person or a report. This means the names of the expert panel members should be listed and the elicitation(s) associated with each should be explicit. It should be possible for scientific peers to review and, if necessary, reproduce the results of the study, and arrive at the same numerical values, if not the same conclusions. Document the specific process used to elicit judgmental probabilities and communicate the results back to the panel of experts.
Our Example will emphasize the Phase 4 Q&A

11. Example: Estimate Commute Time
Why this example?
Fairly easy to find a subject matter expert
It is a parameter that is measurable
Most experts can estimate a most likely time
Factors that drive uncertainty can be readily identified
People general care about their morning commute time!
Let’s begin with Phase 1 … Motivating the Expert:
Motivating the expert:
Sometimes the expert has no choice (mandatory).
Sometimes expert doesn’t want to disappoint or contradict his/her boss.
Sometimes expert is uncomfortable (or has little faith) in the EE process.
Again, if you send an advance copy, this could reduce the time to go over the Q&A process (and help it go more smoothly)
However, many times the expert is too busy or distracted to take a good look at the read-ahead.
Motivating the expert
Explain the importance & reasons for collecting the data
Explore stake in decision & potential for motivational bias

12. EE Phase 2: Commute Time Structuring objective, assumptions & process
Be explicit about what you want to know & why you need to know it
Clearly define variable & avoid ambiguity and explain data values that are required (e.g. hours, dollars, %, etc)
The Interviewer should have worked with you to develop the Objective and up to 5 Major Assumptions in the table below
Please resolve any questions or concerns about the Objective and/or Major Assumptions prior to continuing to "Instructions".
Objective: Develop uncertainty distribution associated with time (minutes)
it will take for your morning commute starting 1 October 2014.
Assumption 1: Your commute estimate includes only MORNING driving time
Assumption 2: The commute will be analogous to the one you've been doing
Assumption 3 Period of commute will be from 1 Oct 2014 thru 30 Sep 2015
Assumption 4 Do not try to account for extremely rare & unusual scenarios
Assumption 5: Unless you prefer otherwise, time will be measured in minutes
Structuring objective:
Many times the expert already knows the drill. So, in this case, be brief!
While structuring objective, emphasize to expert to:
1. Not try to account for extremely rare & unusual scenarios (e.g. where you wouldn't achieve the objective of getting to work). Example: Plane crashing on 14th Street Bridge year flood.
While at the same time …
2. Not completely ignore scenarios that, although occasional, has some ‘real’ chance (~1-2% chance) of occurring (e.g. 3 accidents along route).

13. EE Phase 3: Commute Time Training (conditioning) the expert
Go over instructions for Q&A process
Emphasize benefits of time constraints & 2 iterations
Instructions: This interview is intended to be conducted in two Iterations.
Each iteration should take no longer than 30 minutes.
A. Based on your experience, answer the 12 question sets below.
B. Once you've completed the questions, review them & take a 15 minute break.
C. Using the triangular graphic to assist you, answer all of the questions again.
Notes:
A. The 2nd iteration is intended to be a refinement of your 1st round answers.
B. Use lessons-learned from the 1st iteration to assist you in the 2nd iteration.
C. Your interviewer is here to assist you at any point in the interview process.
Training (conditioning):
If you send an advance copy, this could reduce the time to go over the Q&A process (and help it go more smoothly)
However, many times the expert is too busy or distracted to take a good look at the read-ahead.

14. EE Phase 3: Commute Time (cont’d)
Training the expert (continued)
For 2 Questions, you’ll need to provide your assessment of likelihood:
Values will be defined by SME

15. EE Phase 4: Commute Time (iteration 1)
Assessing expert’s responses (Q&A) M
PDF created based upon Expert’s responses to Questions 1 through 8.
‘true’ L
Assessing Expert’s Responses (Q&A):
The heart of Expert Elicitation (EE) is the Q&A with the Expert. These are the first 8 of 12 questions.
Expert characterizes input parameter (e.g. WBS4: Commute Time)
Expert estimates Most Likely value, M (note: M might be given)
Expert adjusts M (if applicable) … remind expert of the assumption that Commute is in 2014!
Expert estimates chance the actual value could exceed adjusted M
Expert estimates Lowest value, L (note: L might be given)
Expert estimates chance the actual value could be less than L
Expert estimates Highest value, H (note: H might be given)
Expert estimates chance the actual value could be higher than H
P(x<L)
0.29
P(x>H)
0.10
‘true’ H L H
Given from Expert: L=42, M=55, H=80, p(x<L)=0.29 and p(x>H)=0.10
Calculation of ‘true’ L and H (a) : L = 4.22 and H = … Do these #’s appear reasonable?
(a) Method to solve for L and H presented in “Beyond Beta,” Ch1 (The Triangular Distribution)

16. EE Phase 4: Commute Time (Iteration 1)
Assessing expert’s responses (Q&A)
Given the objective and assumptions …
Characterize input parameter (e.g. WBS4: Commute Time)
What’s the Most Likely value, M?
Adjust M (if applicable)
What’s the chance the actual value could exceed M?
What’s the Lowest value, L
What’s the chance the actual value could be less than L?
What’s the Highest value, H
What’s the chance the actual value could be higher than H?
Assessing Expert’s Responses (Q&A): These questions will be asked while the SME sees a graphic of results!
The heart of Expert Elicitation (EE) is the Q&A with the Expert. These are the first 8 of 12 questions.
Up ahead:
Question 9: Expert creates “value-scale” tailored his/her bias
Question 10: Expert & Interviewer brainstorm risk factors
Question 11: Expert selects top 6 risk factors that contributed to increase in value.
Question 12: Expert scores each risk factor’s “contribution” to parameter uncertainty
After Question 12: The 1st iteration will be complete. Expert will then take a 15 min break then starts at #3 again.
This 1st iteration tends to result in anchoring bias on M, over-confidence on L and H, and poor rationale

17. EE Phase 4: Commute Time (iteration 1)
Question 9: Expert creates “value-scale” tailored his/her bias … What probability would you assign to a value that's "Very Unlikely" What probability would you assign to a value that's "Extremely Unlikely"
Available Selection of Values to the Expert (shaded cells were selected by expert):
Assessing Expert’s Responses (Q&A):
The heart of Expert Elicitation (EE) is the Q&A with the Expert. This is step 9 of the 12 questions.
Expert creates a “value-scale” (values that apply to all questions)

18. EE Phase 4: Commute Time (iteration 1)
Revised Question 9: Expert creates “value-scale” tailored his/her bias … What probability would you assign to a value that's "Very Unlikely" What probability would you assign to a value that's "Extremely Unlikely"
Assessing Expert’s Responses (Q&A):
The heart of Expert Elicitation (EE) is the Q&A with the Expert. This is step 9 of the 12 questions.
Expert creates a “value-scale” (values that apply to all questions)
Pre-defined: %, 50%, 100%
Expert opinion: % & 10%
100% - Expert opinion: 99% & 90%
Indifference 1 = 30% = Linear midpoint between “Very Unlikely” (10%) & Even Chance (50%)
Indifference 2 = 70% = Linear midpoint between “Very Likely” (90%) & Even Chance (50%)
Only 2 probabilities needed to be elicited in order to create a Value-Scale that has 9 categories!

19. EE Phase 4: Commute Time (iteration 1)
Question 10: Expert & Interviewer brainstorm risk factors …
What risk factors contributed to the uncertainty in your estimate?
Create Risk Breakdown Structure (RBS)
Question 11: Expert selects top 6 risk factors …
What are the top 6 risk factors that contributed to your estimate uncertainty?
Assessing Expert’s Responses (Q&A):
The heart of Expert Elicitation (EE) is the Q&A with the Expert. These are steps 10 &11 of the 12 questions.
Expert reviews Triangular Distribution & brainstorms all possible risk factors
This process should help you and her create a rough Risk Breakdown Structure (RBS) by start of second iteration
Expert selects top 6 risk factors that contributed to value uncertainty (i.e. increase in value)

20. EE Phase 4: Commute Time (iteration 1)
Question 12: Expert scores each risk factor’s contribution to uncertainty …
Score each risk factor a value based upon the following instruction:
Expert provides a score for each risk factor (rationale not shown).
Assessing Expert’s Responses (Q&A):
The heart of Expert Elicitation (EE) is the Q&A with the Expert. This is step 12 of the 12 questions.
Expert scores each risk factor’s “contribution” to parameter uncertainty
1st iteration is complete. Expert takes 15 min break then starts at #3 again.
The 1st iteration of Q&A is complete. Recommend the expert take a 15 minute break before re-starting Q&A
Slide 20

21. EE Phase 4: Commute Time (iteration 2)
Assessing expert’s responses (Q&A) M
PDF created based upon Expert’s responses to Questions 3 through 8.
P(x>H)
0.29
Assessing Expert’s Responses (Q&A):
The heart of Expert Elicitation (EE) is the Q&A with the Expert.
We are now in iteration #2 of the Q&A. Questions 1 &2 (from iteration #1) are no longer required.
These are the first 6 of 10 questions:
Expert may adjust Most Likely value, M (note: M might be given)
Expert estimates chance the actual value could exceed adjusted M
Expert estimates Lowest value, L (note: L might be given)
Expert estimates chance the actual value could be less than L
Expert estimates Highest value, H (note: H might be given)
Expert estimates chance the actual value could be higher than H
‘true’ L
‘true’ H
P(x<L)
0.01 L H
Given from Expert: L=40, M=55, H=90, p(x<L)=0.10 and p(x>H)=0.29
Calculation of ‘true’ L and H (a) : L = and H = … Do these #’s appear reasonable?
(a) Method to solve for L and H presented in “Beyond Beta,” Ch1 (The Triangular Distribution)

22. EE Phase 4: Commute Time (Iteration 2)
Assessing expert’s responses (Q&A)
Given the objective, assumptions & input parameter (WBS4):
Do you want to adjust your Most Likely Value, M?
What’s the chance the actual value could exceed M?
Assuming best case: weather, accidents, road const, departure time, etc.:
What’s the Lowest value, L
What’s the chance the actual value could be less than L?
Assuming worst case: weather, accidents, road const, departure time, etc.:
What’s the Highest value, H
What’s the chance the actual value could be higher than H?
Note: M, L or H might already be given. Therefore, no expert judgment needed.
As the normative expert (i.e. you) go through this 2nd iteration be sure to do the following:
1. When visiting his/her Lowest value, FIRST talk with expert what “best case” means to him/her:
Best weather = no rain, accidents = 0 or 1 accident? Road construction?
Would you even be allowed to 0400 hrs (i.e. office isn’t open until 6AM, are red lights timed, etc.)
Take notes on these factors affecting “L” because it will help improve RBS and expert’s rationale.
2. When visiting his/her Lowest value, FIRST talk with expert what “worst case” means to him/her:
Worst weather = ice, 3 accidents, road construction along a 3-mile stretch can cause 4 lanes to go to 1 lane!
Would leaving at 0830 hrs mean more traffic? How does feeling tired affect expert’s driving? Is “catching” 1/2 the red lights worst case?
Take notes on these factors affecting “H” because it will help improve RBS and expert’s rationale.
This 2nd iteration helps “condition” expert to reduce anchoring bias on M, counter over-confidence on L and H, calibrate ‘values’ & improve rationale.

23. EE Phase 5: Commute Time (iteration 2)
Verifying encoded values & documentation
Triangular PDF from Iteration 1
Triangular PDF from Iteration 2
L = H =
L = H =
Inputs not necessarily sensitive to risk factors => Optimistic Bias
Inputs sensitive to weighted risk factors => Minimum-Bias
The 2nd iteration helped elicit an L that seems feasible and an H that accounts for worst-case risk factors

24. Results (Triangular & Beta-PERT)
In most cases, Beta-PERT is preferred (vs triangular)
Beta-PERT’s mean is only slightly greater than its mode
However, triangular would be preferred (vs Beta-PERT) if elicited data seems to depict over-confidence (e.g. H value is optimistic)
Triangular PDF compensates for this by ‘exaggerating’ the mean value

25. Conclusion We provided an expert elicitation overview that …
Demonstrated a way to model expert opinion as a triangular distribution
A process that does not “over-burden” the subject matter expert
Incorporated techniques to address expert bias
Iterative Q&A process that includes use of visual aids
Relied on at least a 2nd iteration to help minimize inaccuracy & bias
Convert Triangular to Beta-PERT (if overconfidence was addressed)
Structured the process to help justify expert’s inputs
Rationale required for each response
RBS to help identify what risk factors contribute to uncertainty
Weight risk factors to gain insight as each risk factor’s relative contribution to uncertainty (cost, schedule, etc.,)

26. Potential Improvements
More upfront work on “Training” Expert
Criteria when to elicit mean or median (vs mode)
Add 2 questions to create Modified Beta-PERT
Improve scaling tables for expert opinion
Create “starter” Risk Breakdown Structures”
Facilitates brainstorming process of possible risk factors
Improve method of weighting risk factors
Explore other distributions, e.g. Weibull & LogNormal
Incorporate methods to combine expert opinions
“Training” expert:
What is “most likely” … it’s the mode. What is mean? How is this different than the mode? Etc.
Criteria when to elicit mean or median (vs mode):
Example: Sometimes SME “intuitively” estimates mean for given data type, not mode
Include questions to create Modified Beta-PERT: That is … the factor on “M” doesn’t necessarily have to be = 4
Improve scaling tables for expert opinion: Example: Increase # of categories, have a non-linear scale, etc.
To improve method of weighting risk factors, could use pair-wise comparison
So … hopefully … this adds to the conversation on how best to leverage expert opinion in the cost community …

27. Intuition versus Analysis
Quickly answer the question: “A bat and a ball cost $ 1.10 in total. The bat costs $1 more than the ball. How much does the ball cost?.”
Answer:
Ball cost = $0.05
Bat cost = $1.05
That is, the bat costs $1.00 more than the ball!
Many will INCORRECTLY answer:
Ball cost = $0.10
Bat cost = $1.00
In this case, the bat cost is $0.90 more than the ball 

28. Sources not Referenced in Presentation
Liu, Y., “Subjective Probability,” Wright State University.
Kirkebøen, G., “Decision behaviour – Improving expert judgement, 2010
Vose, D., Risk Analysis (2nd Edition), John Wiley and Sons, 2004
“Expert Elicitation Task Force White Paper,” US EPA, 2009
Clemen, R.T. and Winkler, R.L. (1990) Unanimity and compromise among probability forecasters. Management Science

29. A Step-Wise Approach to Elicit Triangular Distributions
Formerly entitled “An Elicitation Method to Generate Minimum-Bias Probability Distributions”
Questions?
Marc Greenberg

30. Probability Distributions
Bounded
Triangular & Uniform
Histogram
Discrete & Cumulative
Beta & Beta-PERT
Non-Parametric Distributions: Mathematics defined by the shape that is required. Empirical, intuitive and easy to understand.
Unbounded
Normal & Student-t
Logistic
Parametric Distributions: Shape is born of the mathematics describing theoretical problem. Model-based. Not usually intuitive.
Left bounded
Lognormal
Weibull & Gamma
Exponential
Chi-square
Note 1: This is not an exhaustive list of probability distributions (also referred to as continuous distributions).
Note 2: Distributions in red font are commonly used by cost estimators to model uncertainty.
What are Pros & Cons of Bounded? Of Left Bounded? Of Unbounded?
Why prefer “Non-Parametric” distributions for modeling expert opinion?
Overview of Parametric Distributions:
Beta & Beta-PERT … we’ll cover in detail on slides 9 & 10
Normal & Student-t:
Logistic :
Weibull & Gamma defined by alpha and beta parameters:
- Weibull used more & more in cost estimating … Example: it can model extreme values well; used to model RDT&E profiles (Rayleigh(b)).
- Gamma used to model time required for alpha events to occur (with a mean time between events of beta)
Exponential: Defined by beta parameter. Many times used to model Poisson process (e.g. model failure rate of elec circuit, decay of a particle, etc.)
Chi-square: Commonly used to check goodness-of-fit of a statistic to available data
Of the many probability distributions out there, Triangular & Beta-PERT are among the most popular used for expert elicitation

31. Reasons For & Against Conducting EE
Reasons for Conducting an Expert Elicitation
The problem is complex and more technical than political
Adequate data (of suitable quality and relevance) are unavailable or unobtainable in the decision time framework
Reliable evidence or legitimate models are in conflict
Qualified experts are available & EE can be completed within decision timeframe
Finances and expertise are sufficient to conduct a robust & defensible EE
Reasons Against Conducting and Expert Elicitation
The problem is more political than technical
A large body of empirical data exists with a high degree of consensus
Findings of an EE will not be considered legitimate or acceptable by stakeholders
Information that EE could provide is not critical to the assessment or decision
Cost of obtaining EE info is not commensurate with its value in decision-making
Finances and/or expertise are insufficient to conduct a robust & defensible EE
Other acceptable methods or approaches are available for obtaining the needed information that are less intensive and expensive

32. Sources of Cost Uncertainty
Best
Practices
Focus of
Cost Risk
Estimation
Source: “Incorporating Risk,” presentation by J. Hihn, SQI, NASA, JPL, 2004

33. Classic “I Forgots” Review preparation Documentation
Fixing Anomalies and ECR’s
Testing
Maintenance
Basic management and coordination activities
CogE’s do spend time doing management activities
Mission Support Software Components
Development and test environments
Travel
Training
Source: “Incorporating Risk,” presentation by J. Hihn, SQI, NASA, JPL, 2004

34. Some Common Cognitive Biases
Availability
Base judgments on outcomes that are more easily remembered
Representativeness
Base judgments on similar yet limited data and experience. Not fully considering other relevant, accessible and/or newer evidence
Anchoring and adjustment
Fixate on particular value in a range and making insufficient adjustments away from it in constructing an uncertainty estimate
Overconfidence (sometimes referred to as Optimistic bias)
Strong tendency to be more certain about one’s judgments and conclusions than one has reason. Tends to produce optimistic bias.
Control (or “Illusion of Control”)
SME believes he/she can control or had control over outcomes related to an issue at hand; tendency of people to act as if they can influence a situation over which they actually have no control.
• Availability Bias. This results from over-emphasizing the most recent, most easily recalled, or most vivid evidence. For example, if the team just
discussed a catastrophic failure case history, there could be a tendency for the team to assign a higher likelihood to that failure mechanism. This may be appropriate if relevant similar conditions exist between the failure case history and dam being studied, but may not be appropriate if the
similarities are questionable. A facilitator can try to point out the relevance of the latest information and counter it with an opposing view, if
it is not thought to be representative of the issue at hand.
• Representativeness Bias. This results from overemphasizing similarities and neglecting other information. The probability of B, given A is not equal to the probability of A, given B. For example, if slides are observed on steep slopes, one might conclude that steep slopes cause slides, or you are likely to find slides in areas of steep slopes. In reality, it is telling us the opposite; that you are likely to find steep slopes in areas of sliding
(Vick, 2002). While this distinction may seem unimportant, it can make a difference in the likelihood. The fact that slides occur infrequently on flat
slopes might simply be that there aren’t any flat slopes. We also don’t know anything about the false positives, or steep slopes that are stable.
This one is difficult to identify and correct, but if recognized by a facilitator, the appropriate questions can be asked (e.g. “what do we know
about steep slopes that are stable?”).
• Anchoring & Adjustment Bias. This is the tendency for estimates to not vary much from values that are initially presented to the group, either from base frequency information or other sources. This is not mentioned to discourage presentation of base-frequency information, which is often
useful to arrive at a reasonable estimate. However, if this information is presented, it should be as appropriate as possible to the problem being
estimated and consideration should be given as to how the specific case might vary from the general population. Anchoring bias can be avoided to
some extent by discussing potential extreme values before settling on a “best estimate”.
Overconfidence Bias. Perhaps the most pervasive bias is the tendency for people to be more confident than the evidence warrants. This usually
leads to estimates that are closer to zero or to one than they really should be, and/or to distributions that are narrower than they should be.
Unfortunately, the more expertise and knowledge one possesses, the greater the tendency for overconfidence bias appears to be. This tendency
has been demonstrated in experiments involving the general population as well as engineers specifically. One way to help avoid this is to start with
an exercise where estimators need to estimate the likelihood of known events, such as that given in the back of this section. People can then get a feel if they are overconfident.
• Motivational Bias. This results when one or more of the members making estimates have a vested interest in the outcome. For example, if
the designer of the dam is in the room, that person may want to push the view that the dam is well designed and there should be no problems with it, and hence will provide low failure probability estimates. Or perhaps an office needs work, and their representative is quick to push for higher failure probability estimates. A facilitator can attempt to bring out opposing views when an opinion appears to be expressed from a
motivational bias.