1. Understanding and Using Uncertainty Information in Weather Forecasting Susan Joslyn University of Washington

2. Acknowledgements Earl Hunt David Jones Limor Nadav-Greenberg John Pyles Adrian Raftery Karla Schweitzer McLean Slaughter Meng Taing Jeff Thomasson This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745

3. Forecast Uncertainty Available for some time Rarely communicated in public forecasts Underused by weather forecasters

4. Forecast Uncertainty Difficult to understand - Forecasters claim People make mistakes when reasoning with probability Format: Frequency (1 time in 10) is better than Probability (10% chance)

5. Forecast Uncertainty Useful for deterministic forecasts decision? Theoretically Practically useful? It doesn’t matter how good the information if people can’t or won’t make use of it.

6. Goals for Psychology Team Establish uncertainty information is useful Threshold forecast (forecasters & general public) - high wind advisory for boater safety What is best presentation format to enhance Understanding? Decisions?

7. Three Major Lines of Inquiry 1. Does probability information improve threshold forecast? Study 1 2. Does display format (visualization) matter? Study 2 3. Does the wording matter? Studies 3-4 (probability/ frequency)

8. Study1 Does Probability Information Improve Threshold Forecast? Participants: Advanced atmospheric science students Task: Forecast wind speed and direction Decide whether to issue high wind advisory (winds > 20 knots)

9. Within Subject Design Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) + Chart showing probability of winds > 20 k Condition 1 Condition 2

10. Probability of Winds ≥ 20k

11. Within Subjects Design Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) + Chart showing probability of winds > 20 k Condition 1 Condition 2 Same participants, same weather Only difference is probability product

12. Results Threshold Forecast: People posted fewer wind advisories with probability product. Similar ability to discriminate between high wind and low wind event (sensitivity).

13. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

14. Conclusion: Uncertainty Information IS Beneficial for Threshold Increased advisories when high winds were very likely Decreased advisories when high winds were unlikely-fewer false alarms Increase trust in warnings!

15. Study 2 Does Display Format Matter? 3 different visualizations of 90% predictive interval Range of likely wind speeds All conditions included median wind speed chart deterministic forecast

16. 3 Visualizations: Between subjects 1. 90% Upper bound: warmer colors = higher wind speed “ observed wind speeds will be higher only 1 time in 10” worse case scenario: highest likely winds

17. 3 Visualizations 1. 90% Upper bound: wind speeds will be higher only 1 time in 10 warmer colors = higher wind speed 2. Margin of error: range of wind speeds between UB & median display of uncertainty in the forecast warmer colors = more uncertainty

18. 3 Visualizations 1. 90% Upper bound: wind speeds will be higher only 1 time in 10 warmer colors = higher wind speed 2. Margin of error: range of wind speeds between upper bound and median warmer colors = more uncertainty 3. Box plot: median 90% Upper bound 90% lower bound Wind Speed in knots Wind speed in knots

19. Method Participants: Atmospheric Science students (replicated on NOAA Forecasters) Practice: Learned how to read charts Test: - Forecast wind speeds - Threshold: high wind advisory (winds >20 knots) - Rate uncertainty in forecast

20. Results: Wind Speed Forecast UB forecast significantly higher wind speeds Display provided a high anchor (Tversky & Kahneman, 1982) Box Plot Upper bound Margin of Error Knots above the Median 1.55 2.02 1.17

21. Results: High Wind Advisories Likelihood of high winds Box Plot Upper Bound Margin of Error HIGH Median > 20K 98.44% 94.45% 91.67% MEDIUM Median 15-20K 32.40% 31.24% 27.95% LOW Median <15 K 3.57% 3.97%2.38% People in the box plot condition: posted significantly more advisories most in high likelihood situations

22. Results: Uncertainty Rating Box plot.81 Upper Bound.89 Margin of Error.97 MoE best for detecting relative uncertainty They learned: “The wider the range the greater the uncertainty” Ratings in the MoE significantly more highly correlated to range correlation

23. Conclusion: Format Matters Box Plot better threshold forecast wind speed: no bias (salient high and low anchors) MoE detect relative uncertainty in forecast Upper higher winds speeds: bias (anchor) Bound no benefit to threshold forecast

24. Study 3 & 4 Does Wording Matter? Participants: Psychology undergraduates Frequency is easier to understand than probability (Gigerenzer, 1995, 1999, 2000) –Research on complex problems –Is that true of simple expressions of uncertainty?

25. Does Wording Matter? There is a 10% chance that the wind speeds will be greater than 20 knots.

26. Method Procedure: Fill out questionnaire rating expressions of uncertainty Decide whether or not to post a high wind advisory Suppose that there is a 10% chance that the wind speeds will be greater than 20 knots. “ How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble) ” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

27. Method Procedure: Fill out questionnaire rating expressions of uncertainty Decide weather to post a wind advisory Suppose that there is a 10% chance that the wind speeds will be greater than 20 k. “ How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble) ” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

28. Method Procedure: Filled out questionnaire rating expressions of uncertainty Decide weather to post a wind advisory Suppose that 1 time in 10 the wind speeds will be greater than 20 knots. “ How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble) ” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

29. Study 3 2 Variables: Wording & Likelihood Probability Frequency 10% chance = 1 time in 10 90% chance = 9 times in 10

30. Study 3: Likelihood of High Wind Held Constant 1 time in 10 wind speeds = 9 times in 10 wind speeds will be greater than 20 knots will be less than 20 knots

31. Results: Reversal Error Rate from wrong side of scale Suppose that there is a 90% chance that the wind speeds will be less than 20 knots. “ How likely are the wind speeds to be less than 20 knots? (please fill in a bubble) ” O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O They completely misunderstand the phrase Most in “90% (9 in 10) less than” wording Which is it? High likelihood? Less than? Reversal error

32. Study 4 Manipulated Less / Greater Less Greater 10% chance less10 % chance greater

33. Added 2 levels of likelihood Less Greater 10% chance less10 % chance greater 1 in 10 less1 in 10 greater 30% chance less30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater

34. Equivalent Expressions Less Wording Greater Wording 10% chance less10 % chance greater 1 in 10 less1 in 10 greater 30% chance less30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater

35. Equivalent Expressions Less Wording Greater Wording 10% chance less10 % chance greater 1 in 10 less1 in 10 greater 30% chance less30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater

36. Results: Reversal Error More often in “less than” wording (4x as likely) Mean reversal error per person Less than.41 Greater than.10 High vs. low likelihood does not matter Frequency wording does not help

37. Results: Wind Advisories 10% 30%70% 90%

38. Results: Wind Advisories 10% 30%70% 90%

39. Results: Wind Advisories 10% 30%70% 90%

40. Results: Probability “less” is worst 10% 30%70% 90% Reversal error subjects eliminated from this analysis 10%30%70% 90%

41. Conclusion: Wording Matters “Less than” wording is difficult (reversal errors) Wind speed advisories in “probability less” - too many advisories in low ranges - too few in high ranges Frequency protects against posting errors generated by “less than” wording

42. Conclusions Probability information improves threshold forecasts –Many end-user weather decisions are yes/no threshold decisions The right display format –Improves understanding MoE communicates relative uncertainty –Improves weather decisions Box Plot increases warnings in high likelihood Box Plot unbiased wind speed forecast Wording matters –“Less than” is confusing –Frequency helps sometimes NOT in reversal errors HELPS in posting advisories

43. The End

44. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

45. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

46. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

47. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

48. Study 1: Rating 10% was rated significantly higher Probability condition: 10% chance (M=1.32) 90% chance (M=.99) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Frequency condition: 1 in ten (M=1.06) 9 out of 10 (M=.98) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

49. Study 2: Rating 10 was rated higher--did not reach significance 10% (1 in 10) greater (M=1.25) 90% (9 in 10)less (M=.97) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O 10% (1 in 10) less (M=.98) 90% (9 in 10)greater (M=.88) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

50. Study 1: Reversal Error Mean reversal error per person 90% (9 times) less than.83 10% (10 times) greater than.33

51. User Needs & Understanding Naval Forecasters Terminal Aerodrome Forecast (TAF) posted at regular intervals while fulfilling other duties

52. Microphone & recorder Method Talk-aloud while creating TAF

53. Numerical Model: MM5 Satellite Synoptic Pattern Comparison 1. Compare position of low in the model & satellite 2. Assess differences in movement and position 3. Adjust forecast accordingly

54. Compare Predicted to Observed Values 1. Access NOGAPS predicted pressure for current time 29.69 2. Access current local pressure and 29.69 subtract from NOGAPS - 29.64.05 3. Access NOGAPS predicted pressure for 29.59 forecast period and subtract error amount -.05 4. Forecast 29.54

55. Results Naval forecasters rely heavily on models (1/3-1/2 source statements referred to models) Statements implying understanding of model uncertainty Model biases and strengths Initialization of model run Strategies for determining uncertainty Evaluation of degree of uncertainty Adjusting model predictions

56. Conclusions Uncertainty? Error in deterministic forecast? Subsequent questionnaire study: confidence is related to their assessment of model performance

57. Probability Problem The probability that a woman getting a mammogram has breast cancer is 1%. If the woman has breast cancer the probability is 80% that she will have a positive mammogram. If the woman does not have breast cancer the probability that she will still have a positive mammogram is 10%. You have a patient that has a positive mammogram (no symptoms)--what is the probability she has breast cancer.

58. Frequency Problem Ten out of every 1,000 women have breast cancer Of those 10 women with breast cancer 8 will have a positive mammogram Of the remaining 990 women without breast cancer, 95 will still have a positive monogram You have a sample of women who have positive mammograms in your screening (no symptoms) How many of these women will actually have breast cancer?

59. Results: Probability “less” is worst Reversal error subjects eliminated from this analysis 10%30%70% 90%