Creating probability forecasts of binary events from ensemble predictions and prior information - A comparison of methods Cristina Primo Institute Pierre Simon Laplace (IPSL) Ian Jolliffe, Chris A. T. Ferro and David B. Stephenson Climate Analysis Group Department of Meteorology University of Reading

2 Outline: How to improve probabilistic forecasts of a binary event? - Use prior information: Bayesian methods. - Calibrate the model: Logistic regression. Illustration of methods with an example: - 3-day ahead precipitation in Reading (UK) - 5-month ahead forecast of Dec. Niño 3.4 Index Conclusions

3 Aim: Forecast a binary event at a future time 1 if the event is observed to occur 0 otherwise NOTATION: Number of members that forecast the event. Numerical models provide an ensemble of forecasts for time : ensemble size 1 if the -th member forecast the event at time 0 otherwise

4 How to estimate the probability of the event?

5 (a) Temporal series Do not use ensemble forecasts.

6 How to estimate the probability of the event? (b) Frequentist approach Just use ensemble forecasts (do not use past observations).

7 How to estimate the probability of the event? (c) Bayesian approach Use ensemble forecasts and prior information about past data, expert opinion or a combination between them.

8 How to estimate the probability of the event? (d) Calibration approach Incorporate the relationship between past observations and past ensemble forecasts.

9 Easy to obtain. When and, the forecaster issues probabilities of 0 and 1 (event completely impossible or completely certain to occur). There is no estimate of uncertainty on the predicted probability. The probabilities take only a finite set of discrete values. It is unlikely the forecaster really believes this statement !! 1) Frequentist approach The probability is estimated by the relative frequency: = Probability that the event is observed to occur (unknown).

10 2) Bayesian approach : provides us with a posterior distribution of the predicted probability. Estimate a distribution a priori including the uncertainty in the parameters: Model uncertainty of the ensemble forecasts (likelihood) as a conditional distribution: Obtain a posterior distribution (Bayes’ theorem): Davison A. C.,Cambridge University Press (2003) If the model is perfectly calibrated, the probability that an ensemble member forecast the event is also.

11 2) Beta approach Observations Beta(0.5,0.5) Beta(1,1) Beta(2,2) Beta(10,20) Katz and Ehrendorfer, Weather and Forecasting (2005). Forecasts + choose a Prior distribution likelihood p.d.f. Beta( ,  ) Posterior distribution Bayes´ theorem But both  and  are unknown !!

12 How to choose the prior distribution? 2) Calculate: a central point: a measure of the spread: = weight, = Number of past observations The weight gives different importance to prior belief and model forecasts and is chosen to minimize the logarithmic score. Rajagopalan et al. (2002) method is a particular case, where:  =  = 0. This is equivalent to the frequentist approach. = climatology

13 3) Calibration Technique: Both unknown Predictor (given by the ensemble forecasts). Link Function The parameters of the logistic regression are calculated to maximize the likelihood. Logistic regression If is an explanatory variable calculated from the ensemble forecasts, then:

14 Relative frequencies: Logit transformation of the relative frequencies: Include prior information: Which explanatory variable to use? Roulston and Smith, Mon. Wea. Rev. (2002)

15 1)Relative frequencies 2) Beta approach Rajagopalan et al. (2002) 3) Logistic Regression Summary of methods

16 Example: Daily winter precipitation at Reading (UK) Forecasts: 3-day ahead 50-member forecasts of daily total precipitation from Ensemble Prediction System (EPS) at ECMWF for a grid point near Reading (UK) forecast Period: Dec-Jan-Feb from 1997 to n= 812 daily observations m x n = 50 x 812 = forecasts Binary event: precipitation above a threshold. Observations: total daily precipitation observed at the University of Reading atmospheric observatory.

17 Precipitation in Reading: The model is not perfectly calibrated 0.1 mm (WMO def. of wet day) 2 mm (perc= 75.6%) 10 mm (perc.=97%, extreme event).

18 Threshold=0.1mm Threshold=10mm Threshold=2mm The predicted probability can be expressed as a function of the frequency approach: F is a lineal function in the RLZ method and non linear in the Logistic Regression approach.

19 BRIER SCORE0.1 mm2 mm10 mm Climatology Frequencies Bayesian approach (RLZ) Log. Reg. (Xt = qt) Log. Reg. (Xt = logit qt) Brier Score: All the BS improve the frequencies one (  =0.05).

20 Example 2: Niño-3.4 SST index Observations: Niño-3.4 SST index Forecasts: 5-month ahead 9-member ensemble forecasts of Niño 3.4 SST index from the coupled ECMWF model ( DEMETER). Period: hindcasts for December started in the 1st of August of each year from 1958 to n = 44 observations m x n = 9 x 44 = 396 forecasts (Palmer et al. Bull. Am. Meteorol. Soc., 2004) Binary event: Index above the median.

21 Years Observations and Forecasts Niño 3.4 SST index: Observations Forecasts

22 Niño 3.4 SST index: 90% perc. 75% perc. median Median 75% 90% We calibrate the data when we codify them.

23 Brier Score: BRIER SCORE0.1 mm2 mm10 mm Climatology Frequencies Bayesian approach (RLZ) Log. Reg. (Xt = qt)

24 Conclusions based on this example: - Use of prior information via the Beta distribution gives forecasts that have more skill than the frequentist ones - Calibration using logistic regression gives forecasts that have more skill than the frequentist ones - A combination of Beta technique and calibration improves each technique separately. Work is still necessary to choose the best predictor for the logistic regression and the best way to combine both techniques.

25 END

26