# Problems with Understanding Probability Richard Washington (based on Auckland presentation by N Nicholls and the paper:

## Presentation on theme: "Problems with Understanding Probability Richard Washington (based on Auckland presentation by N Nicholls and the paper:"— Presentation transcript:

Problems with Understanding Probability Richard Washington richard.washington@geog.ox.ac.uk (based on Auckland presentation by N Nicholls and the paper: Nicholls, N. 1999: Cognitive illusions, Heuristic and Climate Predictions, BAMS, 80, 7, 1385- 1397)

Problems with Probability and Group Work From the course so far, you can see that probability is an important part of forecasting From your experience in seasonal forecasting you will know that group work and discussions are important to the forecaster and user… What constrains the effective use of forecasts? The literature on how humans make decisions has many lessons for us…..

Impediments to the use of seasonal climate predictions Scientific problems (eg., limited skill) Inappropriate content (eg., categorical forecasts) External constraints (eg., inability of users to change decisions) Complexity of target system (eg., impacts of a predicted climate anomaly may be unpredictable for some sectors) Communication problems (eg., confusion due to multiple forecasts) User resistance or misuse (eg., user conservatism) Cognitive illusions and biases e.g. problems with understanding and communicating probability

Impediments to the use of seasonal climate predictions Cognitive illusions and biases e.g. problems with understanding and communicating probability Consider an imaginary seasonal forecast which is known to be perfect….. Would we be able to persuade users to exploit the information? Would all sectors benefit from the information?

What forecasts (seem to) mean, B.Fischhoff, Int. J. Forecasting, 10, 287-303 (1994) A forecast is just a set of probabilities attached to a set of future events. In order to understand a forecast, all one needs to do is to interpret those two bits of information. Unfortunately, there are problems in communicating each element, so that the user of a forecast understands what its producer means.

Cognitive Illusions… Cognitive illusions are like optical illusions …lead to errors we commit without knowing we are doing so arise from problems in quantifying and dealing with probability, uncertainty and risk Leads to …. –Uncertainties to be denied –Risks to be mismanaged (sometimes overestimated, sometimes underestimated) –Unwarranted confidence Experts and general public both to blame

Some problems with probability Framing effect Availability Anchoring Assymetry between losses and gains Ignoring base rates Overconfidence Decision regret Inconsistent intuition Belief persistence Group conformity Confirmation and hindsight bias

Framing effect The way a problem (or forecast) is posed can influence a decision……………..

Framing effect East Africa faces an outbreak of an unusual disease which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program (A or B) would you favour? –If A is adopted, 200 people will be saved –If B is adopted there is a 1/3 probability that 600 people will be saved and a 2/3 probability that nobody will be saved

Framing effect East Africa faces an outbreak of an unusual disease which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program would you favour? –If A is adopted, 200 people will be saved –If B is adopted there is a 1/3 probability that 600 people will be saved and a 2/3 probability that nobody will be saved Now, which of these alternatives (C or D) would you favour?: –If C is adopted 400 people will die –If D is adopted there is a 1/3 probability that nobody will die, and 2/3 probability that 600 people will die

Framing effect (results of surveys in red) East Africa faces an outbreak of an unusual disease which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program would you favour? –If A is adopted, 200 people will be saved 72% (risk averse) –If B is adopted there is a 1/3 probability that 600 people will be saved and a 2/3 probability that nobody will be saved 28% Which of these alternatives would you favour?: –If C is adopted 400 people will die 22% –If D is adopted there is a 1/3 probability that nobody will die, and 2/3 probability that 600 people will die 78% (risk taking option)

Framing effect (results of surveys in red) East Africa faces an outbreak of an unusual disease which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed. Which program would you favour? –If A is adopted, 200 people will be saved 72% (risk averse) –If B is adopted there is a 1/3 probability that 600 people will be saved and a 2/3 probability that nobody will be saved 28% Which of these alternatives would you favour?: –If C is adopted 400 people will die 22% –If D is adopted there is a 1/3 probability that nobody will die, and 2/3 probability that 600 people will die 78% (risk taking option) But……. A = C And B = D Change in framing leads to change in decision

Framing effect Forecasts given as likelihood of drought may lead to different decisions to forecasts expressed as non-likelihood of wet conditions. So, forecasting a 30% chance of drought is likely to cause a different response to forecasting a 70% chance of normal or wet conditions –Even though the forecasts are the same Very different responses can be initiated by small changes in wording (framing) of a forecast

Availability The availability or prominence of events e.g. in the media can bias our perception. The high profile of the 1982/83 and 1997/98 El Nino event may bias a user to expect each El Nino event to be like these big events e.g. southern Africa 1997/98 event was expected to behave like the 1982/3 event, while other El Nino events were forgotten as a comparison Leads to possibility of overestimating unlikely events –e.g. El Nino events in southern Africa lead to drought…… –But El Nino events have been wet in southern Africa and droughts have happened in without an El Nino………….

El Niño… or is it El Nonesense The experts forecast dry weather, and farmers thought they were prepared for it. But then the rains came… A year ago, climate scientists began reporting that they were observing the strongest El Niño ever recorded, with temperatures in the central, and eastern Pacific two to five degrees above normal. El Niño is an abnormal warming of large parts of the central and eastern Pacific which is believed to set in motion major disruptions of weather around the world. Warned of the coming "mother of all El Niños", Southern Africa braced itself for major drought, since that had been the effect of lesser El Niños in the past, Leonard Unganai, of the Drought Monitoring Centre in Zimbabwe, said recently at the ENRICH Southern African Regional Climate Outlook Forum. ENSARCOF, a body of experts and stakeholders, gathered in the Pilanesberg for a post-mortem of their controversial forecasts for the 1997/1998 summer rainfall season. The ENSARCOF team had first met in Kadoma, Zimbabwe, in September 1997 to issue a sober, scientific consensus forecast of the likely impact of El Niño on the coming summer season and counter the scaremongering. For the first half of the summer (October, November and December), ENSARCOF forecast above normal rainfall for northern Tanzania with about-normal rainfall for October and November in most of the rest of the region, with the possibility of a drier December. They indicated chances of above-normal rain on the western side of the region and below normal on the eastern side. Cautious For December to March, the main rainy season, ENSARCOF forecast above-normal rainfall for north-eastern Tanzania, and parts of Zambia and northem Mozambique, drying further south with "significantly below-normal" rain over South Africa, southem Mozambique, Lesotho and Swaziland. These forecasts were certainly of dry conditions, but they were cautious. Yet, all over the region, contingency measures were taken according to the original perception of a catastrophic drought. As Eugene Poolman, research director of the SA Weather Bureau, reported at Pilanesberg, many small farmers did not plant crops at all. In Zimbabwe, ENSARCOF heard, farmers didn't plant at all or. planted late or less, or turned to shorter-growing crops or less productive and less profitable drought-resistant seeds. Livestock was culled. Many banks refused to grant farmers credit. The Zambian national electricity company made plans to reduce power, water officials planned to sink new boreholes, the water development board stopped licensing water rights, and the food reserve agency made plans to import maize. In the event, the lay prophets of doom were proved to be false prophets, and even the scientists were off the mark. The forecast for the first half of the summer was the more accurate, although less so for the south, including South Africa, Lesotho, Swaziland and southern Mozambique,where some eastern areas had above normal rain. JOHANNESBURG SATURDAY STAR AUGUST 1 1998

Anchoring Anchoring refers to the inability to adjust to numbers which are familiar to us and which we start out using – the numerical problem we are considering has this anchor as its reference point…

Anchoring What are the last 4 digits of your work phone number? Add 3000. Remember the number

Anchoring What are the last 4 digits of your work phone number? Add 3000. Do you think the Nile river is longer or shorter (in km) than this number?

Anchoring What are the last 4 digits of your work phone number? Add 2000. Do you think the White Nile river is longer or shorter (in km) than this number? Now, how long is the White Nile River?

Anchoring How long is the River Nile? Telephone Nrs + 2000 2000-2999 3000-3999 4000-4999 5000-5999 6000-6999 7000-7999 8000-8999 9000-9999 etc

Asymmetry between losses and gains First we are offered a bonus of \$300. Then choose between: Receiving \$100 for sure; or Toss a coin. If we win the toss we get \$200; if we lose we receive nothing.

Asymmetry between losses and gains First we are offered a bonus of \$300. Then choose between: Receiving \$100 for sure; or PREFERRED Toss a coin. If we win the toss we get \$200; if we lose we receive nothing.

Asymmetry between losses and gains First we are offered a bonus of \$300. Then choose between: Receiving \$100 for sure; or PREFERRED Toss a coin. If we win the toss we get \$200; if we lose we receive nothing. This time we are first offered a bonus of \$500. Then choose between: Losing \$100 for sure; or Toss a coin. If we lose we pay \$200; if we win we dont pay anything.

Asymmetry between losses and gains First we are offered a bonus of \$300. Then choose between: Receiving \$100 for sure; or PREFERRED Toss a coin. If we win the toss we get \$200; if we lose we receive nothing. This time we are first offered a bonus of \$500. Then choose between: Losing \$100 for sure; or Toss a coin. If we lose we pay \$200; if we win we dont pay anything. PREFERRED

Asymmetry between losses and gains There are big differences in behaviour when faced with loses compared with gains We are usually conservative when offered gains But we are usually adventurous when facing loses So the threat of a loss can be dangerous in seasonal forecasting because we can be too adventurous

Ignoring base rates / prior probabilties A taxi was involved in a hit and run accident at night. Two taxi companies, the Green and the Blue, operate in the city. 85% of the taxis are Green and 15% are Blue A witness identified the taxi as Blue The witness identifies the correct colour 80% of the time and fails 20% of the time What is the probability that the taxi was Blue?

Ignoring base rates Probability that witness saw Green taxi and called it Blue is 0.85 X 0.20 = 17% of all cases Probability that witness saw a Blue taxi and correctly identified it is 0.15 X 0.80 = 12% of all cases More likely that the witness saw a green taxi and thought it was blue! Bayes theorem

Ignoring base rates When subjects respond to this question, the ignore the base rates (in this case: how many blue or green taxis operate in the city) Instead, they rely on the reliability of the witness

Ignoring base rates Imagine that a model (with an accuracy of 90%) predicts that my farm will be suffering from drought in October-December 2002. Historically there is a 10% chance of drought. The model has no bias (i.e. it will forecast droughts with 10% frequency) What is the chance that October-December 2002 will be dry (given the model forecast is for a dry year)?

Ignoring base rates But the chances that October to December 2002 will be dry (given the model forecast) is only 50% P(Non-Dry Year) = 90% P (Dry Year) = 10% P (Correct forecast) = 90% P (Dry Year/Warning) = P(dry) x P(correct) P (dry) x P (correct) + P (not dry) x P (incorrect) = 0.1 x 0.9 / 0.1 x 0.9 + 0.9 x 0.1 =50% Despite the accuracy of the model, the chances of drought are still 50/50 (a coin toss!)

Overconfidence "We don't like their sound. Groups of guitars are on the way out. --Decca Recording Company executive, turning down the Beatles, 1962 Mugabe will be gone by December 2001 – Morgan Tsvangirai June 2001 "With over fifty foreign cars already on sale here the Japanese auto industry isn't likely to carve out a big share of the market for itself."--Business Week, 1968 The 1997/98 El Nino event will be dry in southern Africa! A La Nina event cannot develop this year – the Pacific is too warm – Mason and Graham, April 1998

Overconfidence Which of these causes of death is most frequent (and how confident are you)? –All accidents, or heart attacks? –murder, or suicide?

Overconfidence Select from each pair the most frequent cause of death (and decide how confident you are): –All accidents, or heart attacks? –Homicide, or suicide? First alternative is selected with great confidence by most respondents. The correct answer is the second in each pair.

Overconfidence Overconfidence is a common problem in forecasting It is important to –consider all the reasons that the forecast could be wrong –Consider alternative forecasts –Appoint someone to argue the opposite case

Overconfidence –Remember: –Nature does not make it a priority for us to understand her……….. –Climate is a complicated system

Inconsistent insight (human judgement) Many decisions are reached by judgement, based on intuition e.g. farmers facing a drought will rely on their own experience in their response But human judgement based on intuition is often poor It is often better to substitute an objective decision making scheme than to use human judgement

Belief persistence Primacy effect : Judging things by the information that we receive first…

Belief persistence Primacy effect : Judging things by what comes first… Asch (1946) gave subjects a list of adjectives describing a person, such as –intelligent, industrious, impulsive, critical, stubborn, envious, or –envious, stubborn, critical, impulsive, industrious, intelligent. The impressions of the person were more favourable given the first list than the second!

Belief persistence These are problems that bias a forecaster or a user of a forecast….. If the first forecast we issue is correct, we are likely to become biased about the forecast skill Forecast producers, faced with forecasts from several models, may give more weight to the first forecast he/she receives Inertia may lead forecasters to ignore evidence which contradicts their belief in an existing forecast

Confirmation bias Can you work out the rule I have in my head for producing 3 numbers? 2 4 6 are 3 numbers that obeys the rule. Suggest other number sequences - I will tell you if they conform to the rule or not. Stop when you think you know the rule.

Confirmation bias Wason (1960): 2 4 6 test. Rule is: Three increasing integers. The human understanding when it has once adopted an opinion draws all things else to support and agree with it (Francis Bacon, 1620)

Group conformity - the Asch experiment Each subject was asked whether the test line was equal in length to line A, B, or C. 99% of subjects answered B. Test line ABC

Group conformity - the Asch experiment Each subject was asked whether the test line was equal in length to line A, B, or C. 99% of subjects answered B. If person in front of subject said A, the error rate increased from 1% to 2%

Group conformity - the Asch experiment Each subject was asked whether the test line was equal in length to line A, B, or C. 99% of subjects answered B. If person in front of subject said A, the error rate increased from 1% to 2% If two people ahead of subject said A, error rate increased to 13%.

Group conformity - the Asch experiment Each subject was asked whether the test line was equal in length to line A, B, or C. 99% of subjects answered B. If person in front of subject said A, the error rate increased from 1% to 2% If two people ahead of subject said A, error rate increased to 13%. If three people ahead of subject said A, error rate increased to 33%.

Group conformity - the Asch experiment Each subject was asked whether the test line was equal in length to line A, B, or C. 99% of subjects answered B. If person in front of subject said A, the error rate increased from 1% to 2% If two people ahead of subject said A, error rate increased to 13%. If three people ahead of subject said A, error rate increased to 33%. If, as well, subject was told a monetary reward for the group as a whole depended on how many members of the group gave the correct answer, the error rate increased to 47%.

Group conformity Forecasters and users (e.g. farmers discuss information in groups) Decisions taken in groups may easily be wrong Group dynamics take over and the group may often reach concensus too quickly Groups tend to support the group leader, regardless of their opinion. We can all think of such examples!

Hindsight We tend to remember past events when we have been correct in our forecasts We tend to forget when we have been wrong This leads to increased confidence which is false e.g. 1997/8 ENSO: Experimental Long Lead Bulletin contained few warnings of El Nino: –But…. –Kerr, R.A. 1998: Models win big in forecasting El Nino, Science, 280, 522-523. –Most model forecasts were very poor…

Reducing the effect of cognitive biases on the use of climate predictions De-bias groups preparing forecasts, to reduce over- confidence Avoid anchoring media reports and thus users (eg., to the 1982/83 El Niño) Determine how users interpret probabilities (worded and numerical) - then use these interpretations in forecast preparation Write forecasts to avoid possible framing biases (eg., include multiple versions of forecasts, framed in different ways) Do not combine forecasts subjectively or intuitively Avoid base rate underestimate bias (eg., include specific base rate information in forecast)

References Nicholls, N., 1999. Cognitive illusions, heuristics, and climate prediction. Bulletin of the American Meteorological Society, 80, 1385-1397. Piattelli-Palmarini, M., 1994. Inevitable illusions. Wiley, 242 pp. Plous, S., 1993. The psychology of judgement and decision making. McGraw-Hill, 302 pp.

Download ppt "Problems with Understanding Probability Richard Washington (based on Auckland presentation by N Nicholls and the paper:"

Similar presentations