Model validation Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015

2 Seasonal Forecasting Using the Climate Predictability Tool Validation “Validation” v “verification”: we validate a model, but verify forecasts. In CPT, “validation” relates to the assessment of deterministic (“best guess”) cross-validated and retroactive predictions.

3 Seasonal Forecasting Using the Climate Predictability Tool Forecasts and observations DiscreteContinuous Deterministic It will rain tomorrow There will be 10 mm of rain tomorrow Probabilistic There is a 50% chance of rain tomorrow There is a p% chance of more than k mm of rain tomorrow

4 Seasonal Forecasting Using the Climate Predictability Tool The graph compares the cross-validated hindcasts (green) with the observed values (red). The observed climatological categories are shown in different colours. Are these good? Cross-validation of MAM rainfall for Thailand, using NIÑO4.

5 Seasonal Forecasting Using the Climate Predictability Tool Forecasts and observations DiscreteContinuous Deterministic It will rain tomorrow There will be 10 mm of rain tomorrow Probabilistic There is a 50% chance of rain tomorrow There is a p% chance of more than k mm of rain tomorrow

6 Seasonal Forecasting Using the Climate Predictability Tool Continuous measures compare the best-guess forecasts with the observed values without regard to the categories. They measure the skill of the deterministic forecasts. Tools ~ Validation ~ Cross-validated ~ Performance measures

7 Seasonal Forecasting Using the Climate Predictability Tool Pearson’s correlation Pearson’s correlation measures association (are increases and decreases in the forecasts associated with increases and decreases in the observations?). It does not measure accuracy. When squared, it tells us how much of the variance of the observations is correctly forecast.

8 Seasonal Forecasting Using the Climate Predictability Tool Pearson’s correlation For the Thailand forecasts, the correlation is 0.37, so about 14% of the variance of the observed rainfall was successfully predicted by the model. But measuring skill in terms of squared anomalies creates a few problems: 1. Sensitivity to extremes 2. Why are squared anomalies meaningful?

9 Seasonal Forecasting Using the Climate Predictability Tool Pearson’s correlation If we forecast March only, there are two extremely wet years (2001, with 280% of the average), which contributes to almost 25% of the variance in the observations. If one can forecast 2001 accurately, then 25% of the variance is already forecast.

10 Seasonal Forecasting Using the Climate Predictability Tool Spearman’s correlation Numerator:? Denominator:? How much of the squared variance of the ranks for the observations can we correctly forecast? Huh? Spearman’s correlation does not have as obvious an interpretation as Pearson’s, but it is much less sensitive to extremes.

11 Seasonal Forecasting Using the Climate Predictability Tool Kendall’s tau Denominator:total number of pairs. Numerator:difference in the numbers of concordant and discordant pairs. Kendall’s correlation measures discrimination (do the forecasts increase and decrease as the observations increase and decrease?). It can be transformed to the probability that the forecasts successfully distinguish the wetter (or hotter) of two observations?

12 Seasonal Forecasting Using the Climate Predictability Tool Kendall’s tau A concordant pair occurs when the largest of two X values corresponds with the largest of two Y values… In which of these two Januaries were the sea temperatures in the NINO3.4 region warmest (when did the stronger El Niño occur)? What is the probability of getting the answer correct? Repeat for all possible pairs of forecasts.

13 Seasonal Forecasting Using the Climate Predictability Tool Error measures compare the best-guess forecasts with the observed values without regard to the categories. They measure the skill of the deterministic forecasts.

14 Seasonal Forecasting Using the Climate Predictability Tool Biases indicate systematic errors in the forecasts.

15 Seasonal Forecasting Using the Climate Predictability Tool Biases Mean bias: Always close to zero for cross-validated forecasts; Indicates ability to forecast shifts in climate for retroactive forecasts; Slightly negative if predictand data are positively skewed. Variance or amplitude bias: Typically very small if skill is low because forecasts always close to the mean If there is no mean or variance bias, the RMSE of the forecasts will exceed that of climatology if the correlation is less than 0.5.

16 Seasonal Forecasting Using the Climate Predictability Tool Biases and GCMS ECHAM4.5 MAM rainfall for Thailand from Feb (purple).

17 Seasonal Forecasting Using the Climate Predictability Tool Biases and GCMS ECHAM4.5 rainfall for 21° – 27°N, 88° – 93°E (Bangladesh)

18 Seasonal Forecasting Using the Climate Predictability Tool Overall errors The MSE is a function of Pearson’s correlation, and the variance and mean biases. As with Pearson’s correlation, the MSE and RMSE are sensitive to the extremes because of the squaring. Mean absolute error is the most intuitive: on average the forecasts are “wrong” by about 55 mm.

19 Seasonal Forecasting Using the Climate Predictability Tool Forecasts and observations DiscreteContinuous Deterministic It will rain tomorrow There will be 10 mm of rain tomorrow Probabilistic There is a 50% chance of rain tomorrow There is a p% chance of more than k mm of rain tomorrow

20 Seasonal Forecasting Using the Climate Predictability Tool Categorical measures measure the skill of the deterministic forecasts with the observations as categories.

21 Seasonal Forecasting Using the Climate Predictability Tool Hit scores convert the forecasts to categories and then compare these with the observed categories. But remember that the category containing the best guess is not the most likely!

22 Seasonal Forecasting Using the Climate Predictability Tool Hit scores The contingency tables are based on cross-validated definitions of the categories and so may not perfectly match implied scores from the graph. Some hits can be expected even with useless forecasts (e.g., guessing, or always forecasting the same outcome… Tools ~ Contingency Tables ~ Cross-validated

23 Seasonal Forecasting Using the Climate Predictability Tool Hit skill scores Simply by guessing or always forecasting one category we would expect to get one-third of the forecasts correct (assuming the categories are equiprobable). The skill score counts how many more forecasts are “correct” than by guessing. Tools ~ Contingency Tables ~ Cross-validated

24 Seasonal Forecasting Using the Climate Predictability Tool Scoring near-misses Scoring one-category errors equally to two-category errors seems to be unsatisfactory. But penalizing for large errors has its own problems.

25 Seasonal Forecasting Using the Climate Predictability Tool Linear error in probability space (LEPS) These weights are defined to ensure that forecasts of climatology AND perpetual forecasts of one category AND random guessing have an expected score of zero. The score gives points for “near-misses”.

26 Seasonal Forecasting Using the Climate Predictability Tool Gerrity score This solution has some simpler properties.

27 Seasonal Forecasting Using the Climate Predictability Tool Measures of discrimination: can the forecasts successfully distinguish different outcomes? The observations are categories, but the forecasts are continuous (except where indicated).

28 Seasonal Forecasting Using the Climate Predictability Tool ROC areas: how frequently can the forecasts successfully distinguish different below-normal from normal and above- normal? Or above-normal from normal and below-normal?

29 Seasonal Forecasting Using the Climate Predictability Tool Discrimination – ROC areas When El Niño occurs, do we forecast warmer temperatures than when it does not occur?

30 Seasonal Forecasting Using the Climate Predictability Tool The same test can be extended to multiple categories – what is the probability that the observation in the higher category can be successfully identified.

31 Seasonal Forecasting Using the Climate Predictability Tool … and to infinite categories (or continuous values). What is the probability that we forecast a higher value when we observe a higher value?

32 Seasonal Forecasting Using the Climate Predictability Tool Continuous scores Correlations Pearson’s:% variance Spearman’s:% variance of ranks Kendall’s:2AFC – probability of successfully identifying warmer / wetter observation Errors Mean bias:unconditional error Variance bias:underestimation of variability RMSE:correlation, mean and variance bias MAE:average error

33 Seasonal Forecasting Using the Climate Predictability Tool Categorical scores Hits Hit score:% correct Hit skill:% correct adjusted for guessing LEPS:adjusts for near-misses Gerrity:adjusts for near-misses Discrimination 2AFC:probability of successfully identifying warmer / wetter category ROC:probability of successfully identifying observation in current category

34 Seasonal Forecasting Using the Climate Predictability Tool Significance testing Tools ~ Validation ~ Cross-validated ~ Bootstrap

35 Seasonal Forecasting Using the Climate Predictability Tool Exercises Assess the quality of your forecast model. Try to describe the quality of the model by interpreting the scores rather than just by listing them.