1. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 1/33 How to Communicate Uncertainties Renate Hagedorn European Centre for Medium-Range Weather Forecasts

2. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 2/33 Motivation Main reasons for not using (probabilistic) predictions in decision-making processes include: forecasts are not accurate enough fluctuation of successive forecasts competing or conflicting forecast information history of previous forecasts not available procedures for acquiring and integrating forecasts into decision-making processes have not been defined external constraints forbid flexible response to forecast info local information may be more important value of forecast has not been demonstrated All forecast system or impact system related impediments

3. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 3/33 Motivation Additionally, non-rational thinking or cognitive illusions affect the optimal use of (probabilistic) forecasts Capability of human mind for solving complex problems is limited compared with the size of problems Lack of objectively rational behaviour in real world Use of simple rules of thumb to simplify decision making Heuristics are often helpful, but can lead to biases, especially in uncertain situations where probabilities are encountered

4. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 4/33 Main messages Nothing is certain In many situations, decisions have to be based on probabilities Interpretation of probabilities is sometimes not straightforward Appropriate presentation can help to make the right decisions

5. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 5/33 The illusion of certainty… …or how we construct a single certainty from uncertain cues Do these two table surfaces have the same area and shape?

6. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 6/33 Understanding uncertainties in the real world Examples of well-known sources of cognitive bias formulating the problem: - probabilities vs. frequencies - the framing effect - the anchoring effect underweighting base rates hindsight and confirmation bias belief persistence: Primacy and inertia effect group conformity and decision regret A practical test… (the Monty Hall Problem) Strategies to reduce impact of cognitive illusions Examples of communication/visualization of probabilities

7. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 7/33 Conditional probabilities Breast cancer screening: The facts: - Probability that a woman aged 40-50 has breast cancer = 0.8% - If a woman has breast cancer, probability of positive test = 90% - If a woman does not have breast cancer, prob. of positive test=7% Imagine a woman with a positive test. What is the probability, that she actually has breast cancer? Solution (with Bayes Theorem): - p(disease) = 0.008 - p(pos|disease) = 0.90 - p(pos| no disease) = 0.07 p(disease) * p(pos|disease) - p(disease|pos) = --------------------------------------------------- p(disease) * p(pos|disease) + p(no disease) * p(pos| no disease) 0.09

8. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 8/33 Frequency formulation Breast cancer screening: The facts: - Probability that a woman aged 40-50 has breast cancer = 0.8% - If a woman has breast cancer, probability of positive test = 90% - If a woman does not have breast cancer, prob. of positive test=7% Solution: 1000 women 8: disease 992: no disease 7: positive 1: negative 69: positive 923: negative p(disease | pos) = 7 / (7+69) 0.09

9. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 9/33 Probabilities vs. frequencies Estimated chances of breast cancer given a positive screening mammogram (from Gigerenzer, 2002)

10. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 10/33 The framing effect The way a problem (or forecast) is formulated can affect a decision Imagine that London faces an unusual disease that is expected to kill 600 people. Two alternative programs to combat disease: - Program A: 200 people will be saved - Program B: 1/3 probability 600 saved, 2/3 probability nobody saved Tests indicate that 72% would select program A (risk-averse) Slightly changed wording: - Program C: 400 people will die - Program D: 1/3 prob. that nobody will die, 2/3 prob. that 600 will die Tests indicate that 78% would select program D (risk-taking)

11. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 11/33 The framing effect in real life Professionals, experienced in decision-making, are still affected E.g., information for doctors: - mortality rate of 7% within 5 years -> hesitant to recommend - survival rate after 5 years of 93% -> more inclined to recommend For weather predictions this suggests different response to forecasts expressed as likelihood of drought or non-likelihood of wet conditions E.g., different response to: 30% chance of drought and 70% chance of normal or wet conditions Worded vs. numerical forecast: - 11% judge forecast rain is likely as poor if it did not rain - 37% judge forecast 70% chance of rain as poor if it did not rain although they associate the word likely with probability of 70%

12. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 12/33 Test your knowledge of history What are the last three digits of your phone number? Range of initial anchorAverage estimate 400 – 599629 600 – 799680 800 – 999789 1000 – 1199885 1200 – 1399988 The correct answer is: A.D. 451 In what year would you guess Attila the Hun was defeated? Do you think Attila the Hun was defeated in Europe before or after that year? Add 400 to this number

13. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 13/33 Underweighting base rates Imagine a climate model (with 90% accuracy) predicts drought Historically, there is 10% chance of drought What is the chance that drought will occur in next season? Solution: 100 seasons 10: drought 90: no drought 9: drought FC 1: no-drought FC 81: no-drought FC 9: drought FC p(drought | drought FC) = 9 / (9+9) = 0.50

14. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 14/33 Underweighting base rates Challenge to convince user that Model was correct 90% of time the probability of a drought next season was only 50% Remember: only for equally likely events, accuracy translates into probabilities Imagine a climate model (with 90% accuracy) predicts drought Historically, there is 10% chance of drought What is the chance that drought will occur in next season?

15. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 15/33 Underweighting base rates Imagine a climate model (with 90% accuracy) predicts warmer than normal conditions There is a 50% chance of above normal What is the chance that warmer than normal conditions will occur? Solution: 100 seasons 50: warmer 50: colder 45: warm FC 5: cold FC 45: cold FC 5: warm FC p(warmer | warm FC) = 45 / (45+5) = 0.90

16. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 16/33 Hindsight and confirmation bias Men mark where they hit, and not where they miss. (Jevons, 1958) After finding out whether or not an event occurred, individuals tend to overestimate the degree to which they would have predicted the correct outcome Reported outcomes seem seem less surprising in hindsight than in foresight Example: El Nino 1997 regarded as stunning success, although only one model was reported in the March 1997 NOAA Long-Lead Forecast Bulletin predicting more than slight warming. Some of the very poor forecasts simply ignored in hindsight. Considerable evidence that people tend to ignore (and not search for) disconfirming information of any hypothesis Introduce double-blind test for model assessment?

17. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 17/33 Belief persistence Primacy and inertia also tend to weight evidence inaccurately. People tend to weight more heavily evidence presented first, e.g. persons described as: - intelligent, industrious, impulsive, critical, stubborn, envious are more favourable perceived than persons described as - envious, stubborn, critical, impulsive, industrious, intelligent Inertia may lead people to ignore evidence that contradicts their prior belief (e.g. that a particular forecast system produces useful forecasts) Forecast producers may not recognise the disparity of model predictions, and instead rely too heavily on a forecast that supports their intuitive understanding of the current state of climate

18. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 18/33 Group conformity The Asch test: Is the test line equal to line A, B, or C? Test Line A B C individual test 1 person in front: A 2 persons in front: A 3 persons in front: A monetary reward error rate1%2%13%33%47%

19. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 19/33 Probabilities in Gambling Monty Hall: Lets Make a Deal - in one of the boxes is a bottle of wine - choose 1, 2, or 3 - after choosing, one of the empty boxes will be opened, so that only one empty and one full box are left - you can choose again (stay with first choice or switch) - what is the best strategy? 123

20. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 20/33 Probabilities in Gambling Monty Hall: Lets Make a Deal - in one of the boxes is a bottle of wine - choose 1, 2, or 3 stay switch 123 123 123

21. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 21/33 Strategies to reduce CI influence Recognition that decision-making is inherently biased Understanding how written forecasts, and numerical probability forecasts are interpreted by potential users Try to reduce impact of cognitive illusions by encouraging forecaster groups to de-bias forecasts by e.g. reducing overconfidence or hindsight bias taking care that media reports and forecasts do not cause anchoring to extreme events (e.g. El Nino 82/83) taking care in wording forecasts to avoid framing avoid intuitive approach when combining forecasts, objective approaches exist and are more successful ensuring that base-rates are not ignored using additional visual aids to convey real levels of skill

22. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 22/33 Transformation of probabilities to words TerminologyLikelihood of the occurrence Virtually certainGreater than 99% Probability Very likelyGreater than 90% Probability LikelyGreater than 66% probability About as likely as not33% to 66% probability UnlikelyLess than 33% probability Very unlikelyLess than 10% probability Exceptionally unlikely Less than 1% probability Table 1: IPCC Likelihood Scale TerminologyLikelihood of the occurrence Extremely likelyGreater than 99% Probability Very likely90%-99% probability Likely70%-89% probability Probably – more likely than not 55%-69% probability Equally likely as not45%-54% probability Possible – less likely than not 30%-44% probability Unlikely10%-29% probability Very unlikely1%-9% probability Extremely unlikelyLess than 1% probability Table 2: Forecast Likelihood Scale

23. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 23/33 Use of colour

24. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 24/33 Visualization of Timeseries

25. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 25/33 Probability Maps (medium range) RR>1mmRR>5mm RR>10mm RR>20mm

26. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 26/33 Summary of probability of 4 events Courtesy:Gjermund Haugen, Magnus Ovhed, met.no

27. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 27/33

28. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 28/33

29. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 29/33

30. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 30/33 Unified Prediction System

31. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 31/33 EPS in the Media German TV Dutch TV high normal low Predictability

32. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 32/33 Summary …in this world there is nothing certain but death and taxes. (Benjamin Franklin) Nothing is certain …the theory of probabilities is at bottom only common sense reduced to calculus. (Pierre-Simon, Marquis de Laplace) In many situations, decisions have to be based on probabilities …math is hard, lets go shopping. (Barbie) Interpretation of probabilities is sometimes not straightforward …solving a problem simply means representing it so as to make the solution transparent. (Herbert A. Simon) Appropriate presentation can help to make the right decisions

33. Training Course 2009 – NWP-PR: How to Communicate Uncertainties 33/33 Further Reading: Nicholls, Neville, 1999: Cognitive illusions, heuristics, and climate predictions. BAMS, 80, 1385 - 1397 Gigerenzer, Gerd et al., 1989: The empire of chance: How probability changed science and everyday life. Cambridge University Press, pp. 340. Gigerenzer, Gerd, Peter M. Todd, and the ABC research group, 1999: Simple heuristics that make us smart. Oxford University Press, pp. 416 Gigerenzer, Gerd, 2002: Reckoning with risk. The Penguin Press, pp. 310 WMO, 2007: Guidelines on communicating forecast uncertainty. WMO/TD No.1422 (WMO website) http://www.cut-the-knot.org/probability.shtml