1. Can we distinguish wet years from dry years?
Simon Mason
Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015

2. The ROC
The ROC answers the question: Can the forecasts distinguish an event from a non-event?
Are we more confident it will be dry when it is dry compared to when it is not?
Do we forecast less rain when it is dry compared to when it is not dry?
Do we issue a higher forecast probability for below-normal when it is below-normal compared to when it is not?

3. ROC Retroactive forecasts of MAM rainfall for Thailand.
Which year are you most confident is a dry year?

4. ROC
The most sensible strategy would be to list the years in order of increasing forecast rainfall.
If the forecasts are good, the “dry” years should be at the top of the list.

5. ROC
For the first guess: Repeat for all forecasts.

6. ROC

7. ROC
Plot the correct scores (hit-rate) against the incorrect (false-alarm rate) scores. We want the correct scores to be larger than the incorrect scores, i.e., for the graph to be above the diagonal.

8. ROC What is the year we are most confident is “below”? Was it “below”?
If so score a hit; if not score a false-alarm.
What is the year we are next most confident is “below”?
Cross-validation of MAM rainfall for Thailand, using Feb NIÑO4.

9. ROC What is the year we are most confident is not “below”?
Was it “below”?
If not we scored a false-alarm; if so we scored a hit.
What is the year we are next most confident is not “below”?
Cross-validation of MAM rainfall for Thailand, using Feb NIÑO4.

10. ROC
The bottom left indicates whether the forecasts with strong indications of dry (or wet) are good. Can they indicate that an event will occur? The top right indicates whether the forecasts with strong indications of not dry (or not wet) are good. Can they indicate that an event will not occur?

11. Relative Operating Characteristics

12. Two-Alternative Forced Choice Test
In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)?
What is the probability of getting the answer correct?
50% (assuming that you do not have inside information about ENSO).

13. Two-Alternative Forced Choice Test
In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)?
What is the probability of getting the answer correct?
That depends on whether we can believe the forecasts. Select the forecast with the highest temperature.

14. Two-Alternative Forced Choice Test
We can ask the same question if the forecasts are probabilistic:
In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)?
What is the probability of getting the answer correct?
That depends on whether we can believe the forecasts. Select the forecast with the higher probability.

15. Two-Alternative Forced Choice Test
Retroactive forecasts of MAM rainfall for Thailand.
How well do the forecasts distinguish “dry” years (driest 20%) from other years?
Do we forecast less rain when it is dry compared to other years?

16. Two-Alternative Forced Choice Test
It is easier to calculate by sorting the forecasts so the driest forecast are at the top. We can then count how many of the non-dry years are lower in the table than the dry years.
For 2010: 14 of the 15 non-dry years have wetter forecasts.
For 1998: 14 of 15
For 1995: 14 of 15
For 2005: 14 of 15
For 1992: 12 of 15
In total: 68 of 75 ≈ 91%.

17. Two-Alternative Forced Choice Test
If the forecasts could perfectly discriminate the dry years, the forecasts would be drier than for all the non-dry years, and the dry years would be listed at the top of the table.
If the forecasts could not discriminate the dry years at all, they would be randomly distributed through the table, and there would be a 50% chance of the forecast being drier than on a non-dry year.

18. ROC
The area beneath the red curve, 0.91, gives us the probability that we will successfully discriminate a “dry” year from a non-dry year. The area beneath the blue curve, 0.85, gives us the probability that we will successfully discriminate a “wet” year from a non-wet year.

19. ROC score
The ROC score indicates how successfully we can distinguish an event (e.g., “below-normal”) from a non-event (“normal” or “above-normal”).
How often do we predict less rain when we observe “below-normal” compared to when we observe “normal” or “above-normal”?
But the predictions could be in probabilities:
How often do we predict a higher chance of below-normal when we observe “below-normal” compared to when we observe “normal” or “above-normal”?
… or in categories:
How often do we predict a drier category when we observe “below-normal” compared to when we observe “normal” or “above-normal”?

20. Summary
The ROC answers the question: Can the forecasts distinguish an event from a non-event?
The graph can help identify conditional skill (e.g., can we forecast wet conditions better than dry conditions?)
It can be used to verify deterministic (discrete and continuous) and probabilistic forecasts.

21. Exercises
Diagnose the quality of your forecast models by analysing the ROC graphs.

22. CPT Help Desk web: iri.columbia.edu/cpt/ @climatesociety
@climatesociety
…/climatesociety