Presentation on theme: "A Step-Wise Approach to Elicit Triangular Distributions"— Presentation transcript:
1 A Step-Wise Approach to Elicit Triangular Distributions SCEA Luncheon Series, Washington Area Chapter of SCEAApril 17, • Arlington, VirginiaA Step-Wise Approach to Elicit Triangular DistributionsPresented by:Marc GreenbergOffice of Program Accountability and Risk Management (PARM)Management Directorate, Department of Homeland Security (DHS)The work in this paper is based upon:Discussions I’ve had with many cost geeks over the years on how best to model the uncertainty of cost, duration, schedule, requirements, system characteristics, etc., when little or no data is availableMy review of a wide variety of literature on expert judgment elicitation and probability distributionsRecent yet VERY preliminary work I’ve completed on modeling the elicitation process in a standardized, repeatable way (in Excel) to produce a triangular and a beta-PERT distributions
2 Risk, Uncertainty & Estimating “It is better to be approximately right rather than precisely wrong.Warren BuffettSaid to be a modern version of what was penned by Carveth Read in 1898 (Ref: Logic, Deductive and Inductive, 1898):“It is better to be vaguely right than exactly wrong.”
3 Outline Purpose of Presentation Background The Uncertainty SpectrumExpert Judgment Elicitation (EE)Continuous DistributionsMore details on Triangular, Beta & Beta-PERT DistributionsFive Expert Elicitation (EE) PhasesExample: Estimate Morning Commute TimeExpert Elicitation (EE) to create a Triangular DistributionWith emphasis on Phase 4’s Q&A with Expert (2 iterations)Convert Triangular Distribution into a Beta-PERTConclusion & Potential Improvements
4 Purpose of Presentation Adapt / combine known methods to demonstrate an expert judgment elicitation process that …Models expert’s inputs as a triangular distribution12 questions to elicit required parameters for a bounded distributionNot too complex to be impractical; not too simple to be too subjectiveIncorporates techniques to account for expert biasA repeatable Q&A process that is iterative & includes visual aidsConvert Triangular to Beta-PERT (if overconfidence was addressed)Is structured in a way to help justify expert’s inputsExpert must provide rationale for each of his/her responsesUsing Risk Breakdown Structure, expert specifies each risk factor’s relative contribution to a given uncertainty (of cost, duration, reqt, etc.)This paper will show one way of “extracting” expert opinion for estimating purposes. Nevertheless, as with most subjective methods, there are many ways to do this.
5 The Uncertainty Spectrum Total Certainty = Complete information All knownSpecific UncertaintyPartial information Known unknownsGeneral UncertaintyTotal Uncertainty = No information Unknown unknownsNo Estimate RequiredNo Estimate PossibleExpert OpinionObjective ProbabilitiesSubjective ProbabilitiesData / KnowledgeReference: Project Management Consulting by AEW Services, 2001Expert opinion is useful when little information is available for system requirements, system characteristics, durations & cost
6 Expert Judgment Elicitation (EE) Source: Making Hard Decisions, An Introduction to Decision Analysis by R.T. Clemen
7 Triangular Distribution Used in situations were there is little or no dataJust requires the lowest (L), highest (H) and most likely values (M)Each x-value has a respective f(x), sometimes called “Intensity” that forms the following PDF:L, M & H are all that’s needed to calculate the Mean and Standard Deviation:L M H
8 Beta DistributionBounded on [0,1] interval, scale to any interval & very flexible shapeb > a > 1, distribution is right skewedCalculated Gamma values using Excel’s GAMMALN function:Most schedule or cost estimates follow right skewed pattern. But how do we know a and b? Answer: Beta-PERT Distribution.Sources: 1. Dr. Paul Garvey, Probability Methods for Cost Uncertainty Analysis, 20002. LaserLight Networks, Inc, “Beta Modeled PERT Schedules”
9 Beta-PERT Distribution Requires lowest (L), highest (H) & most likely values (M)Use L, M and H to calculate mean(m) and standard deviation (s) :Use L, H, m and s To calculate shape parameters, a & b :a and b are needed to define the Beta Function and compute the Beta Probability Density:Beta Probability Density Function(as shown in slide 9):Typically for beta-PERT, lambda = 4Deriving alpha & beta for Beta-PERT: Use the matching moment equations in which the minimum and maximum values are known to calculate shape factors which are consistent with the mean and standard deviation, one can derive alpha and beta.Calculated Gamma values using Excel’s GAMMALN function:Sources: 1. Dr. Paul Garvey, Probability Methods for Cost Uncertainty Analysis, 20002. LaserLight Networks, Inc, “Beta Modeled PERT Schedules”
10 Expert Elicitation (EE) Phases Expert Elicitation consists of five phases: (note that Phases 4 & 5 are iterative)Motivating the expertTraining (conditioning) the expertStructuring objective, assumptions & processAssessing (encoding) expert’s responsesQ&A – Expert’s technical opinion is elicitedQuantitative results w/ documented rationaleVerifying encoded values & documentationMotivating phase. Ironically, helps eliminate motivational bias. The purpose of the first stage is to explain why and how the judgmental probabilities will be elicited, and how the results will be used in risk assessment. At this point the expert team will be informed of the items that will require quantification during the process. It is critical at this stage to ensure that all panel members can "play by the rules".Training phase. Helps reduce unintentional bias. The purpose of the second phase is to encourage the Experts to think critically. The Facilitator will expose the experts to a short training session on concepts, objectives, and methods of eliciting judgmental probability. The Experts need to be cautioned to avoid the common biases and sources of error that people make when trying to quantify probability. The training phase will help the Experts to calibrate their responses by the use of common examples and examples related to their specific expertise.Structuring phase. The Facilitator will work with the panel to review the technical issues to be addressed in the elicitation, refine the issues during discussions, and find an appropriate level of detail. Since people tend to be better at working with decomposed problems rather than systems, the Facilitator will work with the panel to segment the problem into manageable pieces. Additionally the Facilitator will work with the expert panel to decide on the specific parameters for which probabilities need to be assessed.Assessing phase. During this phase each Expert's technical opinion is elicited. The Facilitator should direct the panel through an active approach to elicitation. This is achieved by having each expert associate numbers with descriptive statements, while avoiding intuitive or direct assignment. (See Table 1.)Documentation. To stand the test of defensibility, the results of an expert elicitation should be reproducible, accountable, subject to peer review, neutral, and fair. Expert elicitation should be documented so that it is possible, in principal, to reproduce all the calculations involved the process. This means, calculation models should be fully reported and fully specified, questions asked of the experts should be reported, and responses of the experts should be tabulated. The source of all data and estimates in the study should be traceable to a person or a report. This means the names of the expert panel members should be listed and the elicitation(s) associated with each should be explicit. It should be possible for scientific peers to review and, if necessary, reproduce the results of the study, and arrive at the same numerical values, if not the same conclusions. Document the specific process used to elicit judgmental probabilities and communicate the results back to the panel of experts.Our Example will emphasize the Phase 4 Q&A
11 Example: Estimate Commute Time Why this example?Fairly easy to find a subject matter expertIt is a parameter that is measurableMost experts can estimate a most likely timeFactors that drive uncertainty can be readily identifiedPeople general care about their morning commute time!Let’s begin with Phase 1 … Motivating the Expert:Motivating the expert:Sometimes the expert has no choice (mandatory).Sometimes expert doesn’t want to disappoint or contradict his/her boss.Sometimes expert is uncomfortable (or has little faith) in the EE process.Again, if you send an advance copy, this could reduce the time to go over the Q&A process (and help it go more smoothly)However, many times the expert is too busy or distracted to take a good look at the read-ahead.Motivating the expertExplain the importance & reasons for collecting the dataExplore stake in decision & potential for motivational bias
12 EE Phase 2: Commute Time Structuring objective, assumptions & process Be explicit about what you want to know & why you need to know itClearly define variable & avoid ambiguity and explain data values that are required (e.g. hours, dollars, %, etc)The Interviewer should have worked with you to develop the Objective and up to 5 Major Assumptions in the table belowPlease resolve any questions or concerns about the Objective and/or Major Assumptions prior to continuing to "Instructions".Objective: Develop uncertainty distribution associated with time (minutes)it will take for your morning commute starting 1 October 2014.Assumption 1: Your commute estimate includes only MORNING driving timeAssumption 2: The commute will be analogous to the one you've been doingAssumption 3 Period of commute will be from 1 Oct 2014 thru 30 Sep 2015Assumption 4 Do not try to account for extremely rare & unusual scenariosAssumption 5: Unless you prefer otherwise, time will be measured in minutesStructuring objective:Many times the expert already knows the drill. So, in this case, be brief!While structuring objective, emphasize to expert to:1. Not try to account for extremely rare & unusual scenarios (e.g. where you wouldn't achieve the objective of getting to work). Example: Plane crashing on 14th Street Bridge year flood.While at the same time …2. Not completely ignore scenarios that, although occasional, has some ‘real’ chance (~1-2% chance) of occurring (e.g. 3 accidents along route).
13 EE Phase 3: Commute Time Training (conditioning) the expert Go over instructions for Q&A processEmphasize benefits of time constraints & 2 iterationsInstructions: This interview is intended to be conducted in two Iterations.Each iteration should take no longer than 30 minutes.A. Based on your experience, answer the 12 question sets below.B. Once you've completed the questions, review them & take a 15 minute break.C. Using the triangular graphic to assist you, answer all of the questions again.Notes:A. The 2nd iteration is intended to be a refinement of your 1st round answers.B. Use lessons-learned from the 1st iteration to assist you in the 2nd iteration.C. Your interviewer is here to assist you at any point in the interview process.Training (conditioning):If you send an advance copy, this could reduce the time to go over the Q&A process (and help it go more smoothly)However, many times the expert is too busy or distracted to take a good look at the read-ahead.
14 EE Phase 3: Commute Time (cont’d) Training the expert (continued)For 2 Questions, you’ll need to provide your assessment of likelihood:Values will be defined by SME
15 EE Phase 4: Commute Time (iteration 1) Assessing expert’s responses (Q&A)MPDF created based upon Expert’s responses to Questions 1 through 8.‘true’LAssessing Expert’s Responses (Q&A):The heart of Expert Elicitation (EE) is the Q&A with the Expert. These are the first 8 of 12 questions.Expert characterizes input parameter (e.g. WBS4: Commute Time)Expert estimates Most Likely value, M (note: M might be given)Expert adjusts M (if applicable) … remind expert of the assumption that Commute is in 2014!Expert estimates chance the actual value could exceed adjusted MExpert estimates Lowest value, L (note: L might be given)Expert estimates chance the actual value could be less than LExpert estimates Highest value, H (note: H might be given)Expert estimates chance the actual value could be higher than HP(x<L)0.29P(x>H)0.10‘true’ HLHGiven from Expert: L=42, M=55, H=80, p(x<L)=0.29 and p(x>H)=0.10Calculation of ‘true’ L and H (a) : L = 4.22 and H = … Do these #’s appear reasonable?(a) Method to solve for L and H presented in “Beyond Beta,” Ch1 (The Triangular Distribution)
16 EE Phase 4: Commute Time (Iteration 1) Assessing expert’s responses (Q&A)Given the objective and assumptions …Characterize input parameter (e.g. WBS4: Commute Time)What’s the Most Likely value, M?Adjust M (if applicable)What’s the chance the actual value could exceed M?What’s the Lowest value, LWhat’s the chance the actual value could be less than L?What’s the Highest value, HWhat’s the chance the actual value could be higher than H?Assessing Expert’s Responses (Q&A): These questions will be asked while the SME sees a graphic of results!The heart of Expert Elicitation (EE) is the Q&A with the Expert. These are the first 8 of 12 questions.Up ahead:Question 9: Expert creates “value-scale” tailored his/her biasQuestion 10: Expert & Interviewer brainstorm risk factorsQuestion 11: Expert selects top 6 risk factors that contributed to increase in value.Question 12: Expert scores each risk factor’s “contribution” to parameter uncertaintyAfter Question 12: The 1st iteration will be complete. Expert will then take a 15 min break then starts at #3 again.This 1st iteration tends to result in anchoring bias on M, over-confidence on L and H, and poor rationale
17 EE Phase 4: Commute Time (iteration 1) Question 9: Expert creates “value-scale” tailored his/her bias … What probability would you assign to a value that's "Very Unlikely" What probability would you assign to a value that's "Extremely Unlikely"Available Selection of Values to the Expert (shaded cells were selected by expert):Assessing Expert’s Responses (Q&A):The heart of Expert Elicitation (EE) is the Q&A with the Expert. This is step 9 of the 12 questions.Expert creates a “value-scale” (values that apply to all questions)
18 EE Phase 4: Commute Time (iteration 1) Revised Question 9: Expert creates “value-scale” tailored his/her bias … What probability would you assign to a value that's "Very Unlikely" What probability would you assign to a value that's "Extremely Unlikely"Assessing Expert’s Responses (Q&A):The heart of Expert Elicitation (EE) is the Q&A with the Expert. This is step 9 of the 12 questions.Expert creates a “value-scale” (values that apply to all questions)Pre-defined: %, 50%, 100%Expert opinion: % & 10%100% - Expert opinion: 99% & 90%Indifference 1 = 30% = Linear midpoint between “Very Unlikely” (10%) & Even Chance (50%)Indifference 2 = 70% = Linear midpoint between “Very Likely” (90%) & Even Chance (50%)Only 2 probabilities needed to be elicited in order to create a Value-Scale that has 9 categories!
19 EE Phase 4: Commute Time (iteration 1) Question 10: Expert & Interviewer brainstorm risk factors …What risk factors contributed to the uncertainty in your estimate?Create Risk Breakdown Structure (RBS)Question 11: Expert selects top 6 risk factors …What are the top 6 risk factors that contributed to your estimate uncertainty?Assessing Expert’s Responses (Q&A):The heart of Expert Elicitation (EE) is the Q&A with the Expert. These are steps 10 &11 of the 12 questions.Expert reviews Triangular Distribution & brainstorms all possible risk factorsThis process should help you and her create a rough Risk Breakdown Structure (RBS) by start of second iterationExpert selects top 6 risk factors that contributed to value uncertainty (i.e. increase in value)
20 EE Phase 4: Commute Time (iteration 1) Question 12: Expert scores each risk factor’s contribution to uncertainty …Score each risk factor a value based upon the following instruction:Expert provides a score for each risk factor (rationale not shown).Assessing Expert’s Responses (Q&A):The heart of Expert Elicitation (EE) is the Q&A with the Expert. This is step 12 of the 12 questions.Expert scores each risk factor’s “contribution” to parameter uncertainty1st iteration is complete. Expert takes 15 min break then starts at #3 again.The 1st iteration of Q&A is complete. Recommend the expert take a 15 minute break before re-starting Q&ASlide 20
21 EE Phase 4: Commute Time (iteration 2) Assessing expert’s responses (Q&A)MPDF created based upon Expert’s responses to Questions 3 through 8.P(x>H)0.29Assessing Expert’s Responses (Q&A):The heart of Expert Elicitation (EE) is the Q&A with the Expert.We are now in iteration #2 of the Q&A. Questions 1 &2 (from iteration #1) are no longer required.These are the first 6 of 10 questions:Expert may adjust Most Likely value, M (note: M might be given)Expert estimates chance the actual value could exceed adjusted MExpert estimates Lowest value, L (note: L might be given)Expert estimates chance the actual value could be less than LExpert estimates Highest value, H (note: H might be given)Expert estimates chance the actual value could be higher than H‘true’ L‘true’ HP(x<L)0.01LHGiven from Expert: L=40, M=55, H=90, p(x<L)=0.10 and p(x>H)=0.29Calculation of ‘true’ L and H (a) : L = and H = … Do these #’s appear reasonable?(a) Method to solve for L and H presented in “Beyond Beta,” Ch1 (The Triangular Distribution)
22 EE Phase 4: Commute Time (Iteration 2) Assessing expert’s responses (Q&A)Given the objective, assumptions & input parameter (WBS4):Do you want to adjust your Most Likely Value, M?What’s the chance the actual value could exceed M?Assuming best case: weather, accidents, road const, departure time, etc.:What’s the Lowest value, LWhat’s the chance the actual value could be less than L?Assuming worst case: weather, accidents, road const, departure time, etc.:What’s the Highest value, HWhat’s the chance the actual value could be higher than H?Note: M, L or H might already be given. Therefore, no expert judgment needed.As the normative expert (i.e. you) go through this 2nd iteration be sure to do the following:1. When visiting his/her Lowest value, FIRST talk with expert what “best case” means to him/her:Best weather = no rain, accidents = 0 or 1 accident? Road construction?Would you even be allowed to 0400 hrs (i.e. office isn’t open until 6AM, are red lights timed, etc.)Take notes on these factors affecting “L” because it will help improve RBS and expert’s rationale.2. When visiting his/her Lowest value, FIRST talk with expert what “worst case” means to him/her:Worst weather = ice, 3 accidents, road construction along a 3-mile stretch can cause 4 lanes to go to 1 lane!Would leaving at 0830 hrs mean more traffic? How does feeling tired affect expert’s driving? Is “catching” 1/2 the red lights worst case?Take notes on these factors affecting “H” because it will help improve RBS and expert’s rationale.This 2nd iteration helps “condition” expert to reduce anchoring bias on M, counter over-confidence on L and H, calibrate ‘values’ & improve rationale.
23 EE Phase 5: Commute Time (iteration 2) Verifying encoded values & documentationTriangular PDF from Iteration 1Triangular PDF from Iteration 2L = H =L = H =Inputs not necessarily sensitive to risk factors => Optimistic BiasInputs sensitive to weighted risk factors => Minimum-BiasThe 2nd iteration helped elicit an L that seems feasible and an H that accounts for worst-case risk factors
24 Results (Triangular & Beta-PERT) In most cases, Beta-PERT is preferred (vs triangular)Beta-PERT’s mean is only slightly greater than its modeHowever, triangular would be preferred (vs Beta-PERT) if elicited data seems to depict over-confidence (e.g. H value is optimistic)Triangular PDF compensates for this by ‘exaggerating’ the mean value
25 Conclusion We provided an expert elicitation overview that … Demonstrated a way to model expert opinion as a triangular distributionA process that does not “over-burden” the subject matter expertIncorporated techniques to address expert biasIterative Q&A process that includes use of visual aidsRelied on at least a 2nd iteration to help minimize inaccuracy & biasConvert Triangular to Beta-PERT (if overconfidence was addressed)Structured the process to help justify expert’s inputsRationale required for each responseRBS to help identify what risk factors contribute to uncertaintyWeight risk factors to gain insight as each risk factor’s relative contribution to uncertainty (cost, schedule, etc.,)
26 Potential Improvements More upfront work on “Training” ExpertCriteria when to elicit mean or median (vs mode)Add 2 questions to create Modified Beta-PERTImprove scaling tables for expert opinionCreate “starter” Risk Breakdown Structures”Facilitates brainstorming process of possible risk factorsImprove method of weighting risk factorsExplore other distributions, e.g. Weibull & LogNormalIncorporate methods to combine expert opinions“Training” expert:What is “most likely” … it’s the mode. What is mean? How is this different than the mode? Etc.Criteria when to elicit mean or median (vs mode):Example: Sometimes SME “intuitively” estimates mean for given data type, not modeInclude questions to create Modified Beta-PERT: That is … the factor on “M” doesn’t necessarily have to be = 4Improve scaling tables for expert opinion: Example: Increase # of categories, have a non-linear scale, etc.To improve method of weighting risk factors, could use pair-wise comparisonSo … hopefully … this adds to the conversation on how best to leverage expert opinion in the cost community …
27 Intuition versus Analysis Quickly answer the question: “A bat and a ball cost $ 1.10 in total. The bat costs $1 more than the ball. How much does the ball cost?.”Answer:Ball cost = $0.05Bat cost = $1.05That is, the bat costs $1.00 more than the ball!Many will INCORRECTLY answer:Ball cost = $0.10Bat cost = $1.00In this case, the bat cost is $0.90 more than the ball
28 Sources not Referenced in Presentation Liu, Y., “Subjective Probability,” Wright State University.Kirkebøen, G., “Decision behaviour – Improving expert judgement, 2010Vose, D., Risk Analysis (2nd Edition), John Wiley and Sons, 2004“Expert Elicitation Task Force White Paper,” US EPA, 2009Clemen, R.T. and Winkler, R.L. (1990) Unanimity and compromise among probability forecasters. Management Science
29 A Step-Wise Approach to Elicit Triangular Distributions Formerly entitled “An Elicitation Method to Generate Minimum-Bias Probability Distributions”Questions?Marc Greenberg
30 Probability Distributions BoundedTriangular & UniformHistogramDiscrete & CumulativeBeta & Beta-PERTNon-Parametric Distributions: Mathematics defined by the shape that is required. Empirical, intuitive and easy to understand.UnboundedNormal & Student-tLogisticParametric Distributions: Shape is born of the mathematics describing theoretical problem. Model-based. Not usually intuitive.Left boundedLognormalWeibull & GammaExponentialChi-squareNote 1: This is not an exhaustive list of probability distributions (also referred to as continuous distributions).Note 2: Distributions in red font are commonly used by cost estimators to model uncertainty.What are Pros & Cons of Bounded? Of Left Bounded? Of Unbounded?Why prefer “Non-Parametric” distributions for modeling expert opinion?Overview of Parametric Distributions:Beta & Beta-PERT … we’ll cover in detail on slides 9 & 10Normal & Student-t:Logistic :Weibull & Gamma defined by alpha and beta parameters:- Weibull used more & more in cost estimating … Example: it can model extreme values well; used to model RDT&E profiles (Rayleigh(b)).- Gamma used to model time required for alpha events to occur (with a mean time between events of beta)Exponential: Defined by beta parameter. Many times used to model Poisson process (e.g. model failure rate of elec circuit, decay of a particle, etc.)Chi-square: Commonly used to check goodness-of-fit of a statistic to available dataOf the many probability distributions out there, Triangular & Beta-PERT are among the most popular used for expert elicitation
31 Reasons For & Against Conducting EE Reasons for Conducting an Expert ElicitationThe problem is complex and more technical than politicalAdequate data (of suitable quality and relevance) are unavailable or unobtainable in the decision time frameworkReliable evidence or legitimate models are in conflictQualified experts are available & EE can be completed within decision timeframeFinances and expertise are sufficient to conduct a robust & defensible EEReasons Against Conducting and Expert ElicitationThe problem is more political than technicalA large body of empirical data exists with a high degree of consensusFindings of an EE will not be considered legitimate or acceptable by stakeholdersInformation that EE could provide is not critical to the assessment or decisionCost of obtaining EE info is not commensurate with its value in decision-makingFinances and/or expertise are insufficient to conduct a robust & defensible EEOther acceptable methods or approaches are available for obtaining the needed information that are less intensive and expensive
32 Sources of Cost Uncertainty BestPracticesFocus ofCost RiskEstimationSource: “Incorporating Risk,” presentation by J. Hihn, SQI, NASA, JPL, 2004
33 Classic “I Forgots” Review preparation Documentation Fixing Anomalies and ECR’sTestingMaintenanceBasic management and coordination activitiesCogE’s do spend time doing management activitiesMission Support Software ComponentsDevelopment and test environmentsTravelTrainingSource: “Incorporating Risk,” presentation by J. Hihn, SQI, NASA, JPL, 2004
34 Some Common Cognitive Biases AvailabilityBase judgments on outcomes that are more easily rememberedRepresentativenessBase judgments on similar yet limited data and experience. Not fully considering other relevant, accessible and/or newer evidenceAnchoring and adjustmentFixate on particular value in a range and making insufficient adjustments away from it in constructing an uncertainty estimateOverconfidence (sometimes referred to as Optimistic bias)Strong tendency to be more certain about one’s judgments and conclusions than one has reason. Tends to produce optimistic bias.Control (or “Illusion of Control”)SME believes he/she can control or had control over outcomes related to an issue at hand; tendency of people to act as if they can influence a situation over which they actually have no control.• Availability Bias. This results from over-emphasizing the most recent, most easily recalled, or most vivid evidence. For example, if the team justdiscussed a catastrophic failure case history, there could be a tendency for the team to assign a higher likelihood to that failure mechanism. This may be appropriate if relevant similar conditions exist between the failure case history and dam being studied, but may not be appropriate if thesimilarities are questionable. A facilitator can try to point out the relevance of the latest information and counter it with an opposing view, ifit is not thought to be representative of the issue at hand.• Representativeness Bias. This results from overemphasizing similarities and neglecting other information. The probability of B, given A is not equal to the probability of A, given B. For example, if slides are observed on steep slopes, one might conclude that steep slopes cause slides, or you are likely to find slides in areas of steep slopes. In reality, it is telling us the opposite; that you are likely to find steep slopes in areas of sliding(Vick, 2002). While this distinction may seem unimportant, it can make a difference in the likelihood. The fact that slides occur infrequently on flatslopes might simply be that there aren’t any flat slopes. We also don’t know anything about the false positives, or steep slopes that are stable.This one is difficult to identify and correct, but if recognized by a facilitator, the appropriate questions can be asked (e.g. “what do we knowabout steep slopes that are stable?”).• Anchoring & Adjustment Bias. This is the tendency for estimates to not vary much from values that are initially presented to the group, either from base frequency information or other sources. This is not mentioned to discourage presentation of base-frequency information, which is oftenuseful to arrive at a reasonable estimate. However, if this information is presented, it should be as appropriate as possible to the problem beingestimated and consideration should be given as to how the specific case might vary from the general population. Anchoring bias can be avoided tosome extent by discussing potential extreme values before settling on a “best estimate”.Overconfidence Bias. Perhaps the most pervasive bias is the tendency for people to be more confident than the evidence warrants. This usuallyleads to estimates that are closer to zero or to one than they really should be, and/or to distributions that are narrower than they should be.Unfortunately, the more expertise and knowledge one possesses, the greater the tendency for overconfidence bias appears to be. This tendencyhas been demonstrated in experiments involving the general population as well as engineers specifically. One way to help avoid this is to start withan exercise where estimators need to estimate the likelihood of known events, such as that given in the back of this section. People can then get a feel if they are overconfident.• Motivational Bias. This results when one or more of the members making estimates have a vested interest in the outcome. For example, ifthe designer of the dam is in the room, that person may want to push the view that the dam is well designed and there should be no problems with it, and hence will provide low failure probability estimates. Or perhaps an office needs work, and their representative is quick to push for higher failure probability estimates. A facilitator can attempt to bring out opposing views when an opinion appears to be expressed from amotivational bias.