2. Introduction to Seasonal Climate Prediction Liqiang Sun International Research Institute for Climate and Society (IRI)

3. Weather forecast – Initial Condition problem Climate forecast – Primarily boundary forcing problem

4. Climate Forecasts be probabilistic  ensembling be reliable and skillful  calibration and verification address relevant scales and quantities  downscaling

5. OUTLINE  Fundamentals of probabilistic forecasts  Identifying and correcting model errors  systematic errors  Random errors  Conditional errors  Forecast verification  Summary

6. Fundamentals of Probabilistic Forecasts

7. Basis of Seasonal Climate Prediction Changes in boundary conditions, such as SST and land surface characteristics, can influence the characteristics of weather (e.g. strength or persistence/absence), and thus influence the seasonal climate.

8. January 25, 2006UNAM Influence of SST on tropical atmosphere

11. 30 12 30 24 12 24 10 FORECAST SST TROP. PACIFIC (multi-models, dynamical and statistical) TROP. ATL, INDIAN (statistical) EXTRATROPICAL (damped persistence) GLOBAL ATMOSPHERIC MODELS ECPC(Scripps) ECHAM4.5(MPI) CCM3.6(NCAR) NCEP(MRF9) NSIPP(NASA) COLA2 GFDL Forecast SST Ensembles 3/6 Mo. lead Persisted SST Ensembles 3 Mo. lead IRI DYNAMICAL CLIMATE FORECAST SYSTEM POST PROCESSING MULTIMODEL ENSEMBLING PERSISTED GLOBAL SST ANOMALY 2-tiered OCEAN ATMOSPHERE 30 10 REGIONAL MODELS

12. RSMRSM RSMRSM RSMRSM Contingency tables for 3 subregions of Ceara State at local scales (FMA 1971-2000) OBS Coast BNA B 532 N343 A235 RSMRSM RSMRSM RSMRSM Contingency tables for 3 subregions of Ceara State at local scales (FMA 1971-2000) OBS Coast BNA B 532 N343 A235 RSMRSM RSMRSM RSMRSM Contingency tables for 3 subregions of Ceara State at local scales (FMA 1971-2000) OBS Coast BNA B 532 N343 A235 Probability Calculated Using the Ensemble Mean B o N o A o BfNfAfBfNfAf 503020 333433 152560 Contingency Table

13. RSMRSM RSMRSM RSMRSM Contingency tables for 3 subregions of Ceara State at local scales (FMA 1971-2000) OBS Coast BNA B 532 N343 A235 RSMRSM RSMRSM RSMRSM Contingency tables for 3 subregions of Ceara State at local scales (FMA 1971-2000) OBS Coast BNA B 532 N343 A235 1) Count the # of ensembles in each category, e.g., Total 100 Ensembles, 40 ensembles in Category “A” 35 ensembles in Category “N”, and 25 ensembles in Category “B”. 2) Calibration RSMRSM RSMRSM RSMRSM Contingency tables for 3 subregions of Ceara State at local scales (FMA 1971-2000) OBS Coast BNA B 532 N343 A235 Probability obtained from ensemble spread

14. Example of seasonal rainfall forecast (3-month average & Probabilistic)

15. Why seasonal averages? Rainfall correlation skill: ECHAM4.5 vs CRU Observations (1951-95) Should we only be forecasting for February for SW US & N Mexico?

16. Why seasonal averages? Partial Correlation Maps for Individual Months No independent skill for individual months.

17. Why seasonal averages?

18. Why probabilistic? Observed Rainfall (SON 2004)Model Forecast (SON 2004), Made Aug 2004 RUN #1 RUN #4 Two ensemble members from same AGCM, same SST forcing, just different initial conditions. Units are mm/season

19. Why probabilistic? Observed Rainfall Sep-Oct-Nov 2004 (CAMS-OPI) Model Forecast (SON 2004), Made Aug 2004 1 2 3 4 5 6 7 8 Seasonal climate is a combination of boundary-forced SIGNAL, and chaotic NOISE from internal dynamics of the atmosphere.

20. Why probabilistic? Observed Rainfall Sep-Oct-Nov 2004 (CAMS-OPI) Model Forecast (SON 2004), Made Aug 2004 ENSEMBLE MEAN Average model response, or SIGNAL, due to prescribed SSTs was for normal to below-normal rainfall over southern US/ northern Mexico in this season. Need to also communicate fact that some of the ensemble member predictions were actually wet in this region. Thus, there may be a ‘most likely outcome’, but there are also a ‘range of possibilities’ that must be quantified.

21. Forecast Mean Climate Forecast: Signal + Uncertainty “SIGNAL” The SIGNAL represents the ‘most likely’ outcome. The NOISE represents internal atmospheric chaos, uncertainties in the boundary conditions, and random errors in the models. “NOISE” Historical distribution Climatological Average Forecast distribution Below Normal Above Normal Near-Normal

22. Resolution: Probabilities should differ from climatology as much as possible, when appropriate Reliability: Forecasts should “mean what they say”. Probabilistic Forecasts Reliability Diagrams Showing consistency between the a priori stated probabilities of an event and the a posteriori observed relative frequencies of this event. Good reliability is indicated by a 45° diagonal.

23. Identifying and Correcting Model Errors

24. Optimizing Probabilistic Information Eliminate the ‘bad’ uncertainty -- Reduce systematic errors e.g. MOS correction, calibration Reliably estimate the ‘good’ uncertainty -- Reduce probability sampling errors e.g. Gaussian fitting and Generalized Linear Model (GLM) -- Minimize the random errors e.g. multi-model approach (for both response & forcing) -- Minimize the conditional errors e.g. Conditional Exceedance Probabilities (CEPs)

25. Systematic Spatial Errors Systematic error in location of mean rainfall, leads to spatial error in interannual rainfall variability, and thus a resulting lack of skill locally.

26. Systematic Calibration Errors ORIGINAL RESCALED Dynamical models may have quantitative errors in the mean climate RECALIBRATED ORIGINAL … as well as in the magnitude of its interannual variability. Statistical recalibration of the model’s climate and its response characteristics can improve model reliability.

27. January 25, 2006UNAM before and after statistical correction DJFM rainfall anomaly correlation Reducing Systematic Errors MOS Correction (Tippett et al., 2003, Int. J. Climatol.)

28. Converges like S = Signal-to-noise ratio N = ensemble size “True” rms divide by. N=8N=16 N=24 N=39

29. Fitting with a Gaussian Two types of error: PDF not really Gaussian! Sampling error –Fit only mean –Fit mean and variance Error(Gaussian fit N=24) = Error(Counting N=40 )

30. Minimizing Random Errors Multi-model ensembling Combining models reduces deficiencies of individual models Probabilistic skill scores (RPSS for 2m Temperature (JFM 1950-1995)

31. Reliability! A Major Goal of Probabilistic Forecasts Reliability Diagrams Showing consistency between the a priori stated probabilities of an event and the a posteriori observed relative frequencies of this event. Good reliability is indicated by a 45° diagonal.

32. Benefit of Increasing Number of AGCMs in Multi-Model Combination JAS Temperature JAS Precipitation (Robertson et al. 2004)

33. Correcting Conditional Biases METHODOLOGY

34. Conditional Exceedance Probabilities The probability that the observation exceeds the amount forecast depends upon the skill of the model. If the model were perfect, this probability would be constant. If it is imperfect, it will depend on the ensemble member’s value. Identify whether the exceedance probability is conditional upon the value indicated. Generalized linear models with binomial errors can be used, e.g.: Tests can be performed on  1 to identify conditional biases. If  1 = 0 then the system is reliable.  0 can indicate unconditional bias. (Mason et al. 2007, Mon Wea Rev)

35. Idealized CEPs (from Mason et al. 2007, Mon Wea Rev) PERFECT Reliability Positive skill SIGNAL too strong Positive skill SIGNAL too weak Negative skill NO skill β 1 =0 β 1 <0 β 1 = Clim. β 1 |Clim| β 1 >0

36. Conditional Exceedance Probabilities (CEPs) Standardized anomaly 100% 50% 0% Shift Scale Use CEPs to determine biased probability of exceedance. Shift model-predicted PDF towards goal of 50% exceedance probability. Note that scale is a parameter determined in minimizing the model-CEP slope.

37. Adjustment decreases signal Adjustment increases signal Adjustment increases MSE Adjustment decreases MSE CEP Recalibration can either strengthen or weaken SIGNAL CEP Recalibration consistently reduces MSE

38. Effect of Conditional Bias Correction

39. Forecast Verification

40. Verification of probabilistic forecasts How do we know if a probabilistic forecast was “correct”? “A probabilistic forecast can never be wrong!” As soon as a forecast is expressed probabilistically, all possible outcomes are forecast. However, the forecaster’s level of confidence can be “correct” or “incorrect” = reliable. Is the forecaster over- / under-confident?

41. Forecast verification – reliability and resolution If forecasts are reliable, the probability that the event will occur is the same as the forecast probability. Forecasts have good resolution, if the probability that the event will occur changes as the forecast probability changes.

42. UNAM Reliability diagram

43. Ranked Probability Skill Score (RPSS) RPSS measures the cumulative squared error between the categorical forecast probabilities and the observed category relative to some reference forecast (Epstein 1969). The most widely used reference strategy is that of “climatology.” The RPSS is defined as, where N=3 for tercile forecasts. fj, rj, and oj are the forecast probability, reference forecast probability, and observed probability for category j, respectively. The probability distribution of the observation is 100% for the category that was observed and is 0 for the other two categories. The reference forecast of climatology is assigned to 33.3% for each of the tercile categories.

44. Ranked Probability Skill Score (RPSS) The RPSS gives credits for forecasting the observed category with high probabilities, and also puts penalties for forecasting the wrong category with high probabilities.  According to its definition, the RPSS maximum value is 100%, which can only be obtained by forecasting the observed category with a 100% probability consistently.  A score of zero implies no skill in the forecasts, which is the same score one would get by consistently issuing a forecast of climatology. For the three category forecast, a forecast of climatology implies no information beyond the historically expected 33.3%-33.3%-33.3% probabilities.  A negative score suggests that the forecasts are underperforming climatology.  The skill for seasonal precipitation forecasts is generally modest. For example, IRI seasonal forecasts with 0-month lead for the period 1997- 2000 scored 1.8% and 4.8%, using the RPSS, for the global and tropical (30oS-30oN) land areas, respectively (Wilks and Godfrey 2002).

45. Real-Time Forecast Validation

46. Ranked Probability Skill Score (RPSS) Problem The expected RPSS with climatology as the reference forecast strategy is less than 0 for any forecast that differs from the climatological probability – lack of equitability There are two important implications:  The expected RPSS can be optimized by issuing climatological forecast probabilities.  The forecast may contain some potential usable information even when RPSS is less than 0, especially if the sharpness of the forecasts is high.

47. There is no single measure that gives a comprehensive summary of forecast quality.

48. GHACOF SOND +15% bias because of hedging -5% no skill hedging serious bias -20% good resolution: above-normal +10% below-normal +6% 0% resolution because of large biases weak bias +5 reasonable sharpness sharpest forecasts believable?

49. Summary  Seasonal forecasts are necessarily probabilistic  The models used to predict the climate are not perfect, but by identifying and minimizing their errors we can maximize their utility  The two attributes of probabilistic forecasts are reliability and resolution. Both these aspects require verification.  Skill in seasonal climate prediction varies with seasons and geographic regions - Requires research!