1. Uncertain Judgements: Eliciting experts’ probabilities Anthony O’Hagan et al 2006 Review by Samu Mäntyniemi

2. Contents Fundamentals of probability and judgement The elicitation context The psychology of judgement under uncertainty The elicitation of probabilities Eliciting distributions – General Eliciting and fitting a parametric distribution Eliciting distributions – uncertainty and imprecision Evaluating elicitation Multiple experts Published examples Guidance on best practice

3. Fundamentals of probability and judgement Aleatory and epistemic uncertainty: Aleatory: randomness of a system; coin tossing, “natural” variability of population size Epistemic: uncertainty about a fixed quantity; distance between cities, amount of aleatory uncertainty Probability does not exist No universally “correct” probability Subjective probabilities are not typically pre-formed, they are made in response to elicitation Compared to observed events, experts may predict poorly or very well But this is not the point: the point is to formalise what experts think

4. The elicitation context Best practice: face-to-face 1. Background and preparation 1. Identify the model -> identify variables to elicit 2. Identify and recruit experts 1. Evidence of expertise, willingness, lack of stake in findings 3. Motivating and training experts 1. What will be done with the results? Basic knowledge on probability 4. Structuring and decomposition 1. Structure of the model, review the evidence experts will use 5. The elicitation 1. Make questions to have summaries of a distribution, fit a distribution, check the results with the expert

5. The elicitation context: different roles 1. Decision maker (person or group) 1. The user of the results – the client 2. The substantive expert 1. Their knowledge is to be expressed 3. The statistician 1. Expert in the methodology 4. The facilitator 1. Expert in the process of elicitation, manages the dialogue with the expert

6. The psychology of judgement under uncertainty Humans do not usually act as rational agents “Rational” according to probability and decision theory Use of easily adopted strategies (heuristics) Substantive expertise does not guarantee expertise in probability assessment Several kinds of biases may arise from the way how questions are made Bias? At least the probability assessment becomes different For example, probabilities for cause of death: -“Heart attack” or “Something else” -“Heart attack” or “Accident, cancer or something else” “Bias” can potentially be reduced, but care needs to be taken

7. The elicitation of probabilities What is mathematically equal may not be that psychologically Experts can be taught to make probability assessments Interpretation of verbal expressions is highly variable and context specific Analogy to frequency may often be useful, but care is needed Some methods Direct estimation: ask the probability Response scales: e.g. a line with “impossible” and “certain” in both ends. Probability wheels or pie charts Bets: Would you rather bet on the event A or on having “heads” when tossing a fair coin Distribute a pile of objects to bins

8. Eliciting distributions – General In practice: elicit a small number of summaries and fit a distribution There is no true distribution, the fitted distribution tries to be “good” representation of the expert opinion Feedback is important The fitted distribution implies things the expert did not say Revise the distribution until it is accepted by the expert Univariate distributions Summaries based on probabilities, quantiles, credible intervals Intervals with high probabilities (>0.9) are typically poorly assessed (Based on calibration studies)

9. Eliciting and fitting a parametric distribution I Choice of distribution is to some extent a question of mathematical convenience The more complex the model, the more important this is “Overfitting” is recommended: ask more summaries than is needed to identify a distribution Bisection method: 1. Median? 2. Assume that new information says that the true value is below median -> what is the median of the new distribution? ->25% quantile 3. Assume that new information says that the true value is above median -> what is the median of the new distribution? ->75% quantile Could be continued further

10. Eliciting and fitting a parametric distribution II Interactive computing is almost essential New questions based on earlier answers and immediate feedback Identify apparent inconsistencies and correct them Interactive graphics Many published elicitation methods are devised theoretically and have never been used in practice

11. Eliciting distributions – uncertainty and imprecision Experts’ probabilities are proxies of the actual belief Fitted distribution is a proxy for the probabilities What to do with this uncertainty? Could use upper and lower bounds in sensitivity analysis Probabilistic analysis of probability….? Conceptual swamp?

12. Evaluating elicitation Was the elicitation successful? Should describe what an expert thinks, this is difficult to assess In some cases the true value can be revealed, and probabilities compared e.g. weather forecasting There are many methods to use in the comparison Poor calibration with the true values can mean Poor expert knowledge Poor elicitation of expert knowledge Or both

13. Multiple experts Averaging of experts is seen as the most simple and robust method of combining the expert knowledge Cooke’s method: Average experts Higher weight for better experts Test questions from the same subject area, and determination of weights based on success on those WGBAST method seems to be an extension of Cooke’s method “Problem”: similar experts might result in too much weight on the view that they share Group elicitation might have even greater potential

14. Summary Good introduction to state of the art in elicitation Elicitation of model structure? Lot of space devoted for calibration even though the sensibility of calibration is questioned? State of the art is quite messy field, new methods and theories are likely to evolve FEM could take a leading role in actively utilising and developing the elicitation procedures