1. Antje Weisheimer Meteorological Training Course 27 April 2006 Antje Weisheimer Multi-model ensemble predictions on seasonal to decadal timescales

2. Antje Weisheimer Meteorological Training Course 27 April 2006 Introduction A. Murphy (1993): What is a good forecast? 1.Consistency: correspondence between forecaster‘s best judgement and their forecasts 2.Quality: correspondence between forecasts and matching observations  Multifaceted nature of forecast evaluation  Measure-orineted and distribution-oriented scores 3.Value: benefits realised by decision makers through the use of the forecasts

3. Antje Weisheimer Meteorological Training Course 27 April 2006 Structure 1.The multi-model concept 2.Examples:  DEMETER – multi-model seasonal forecasts  EUROSIP – Operational multi-model seasonal forecasts at ECMWF  ENSEMBLES – multi-model seasonal, interannual and decadal forecasts  Others 3.Summary and outlook

4. Antje Weisheimer Meteorological Training Course 27 April 2006 Sources of uncertainty in dynamical seasonal forecasting  initial conditions limited accuracy of observations  ensemble forecasting technique  boundary conditions soil moisture, sea ice, aerosols  model error model structure complex representation of physical processes in models  combination of different skilful, quasi-independent models into a multi-model ensemble parameterisations unresolved processes  stochastic physical parameterisations, see lecture by Judith Berner physical parameter values not precisely known (eg., cloud related parameters)  perturbed parameter approach (Murphy et al., 2004; Stainforth et al., 2005)  numerical representation resolution, truncation  unknown unknowns ?

5. Antje Weisheimer Meteorological Training Course 27 April 2006 climate system state space verification t=0 t=T model 1 Model 2 The multi-model ensemble concept model 2 model 3

6. Antje Weisheimer Meteorological Training Course 27 April 2006 climate system state space verification t=0 t=T The multi-model ensemble concept multi-model ensemble

7. Antje Weisheimer Meteorological Training Course 27 April 2006 The multi-model ensemble concept: Basic scenarios t=0 t=T A t=0 t=T B t=0 t=T C All single-model ensembles lie below / above the veri- fication  Multi-model is impro- ved because of error cancellation One single-model ensemble provides the best forecast  compared to this, the multi-model can only be worse  however, compared to all other single-model ensembles, the multi- model is still improved The verification lies beyond all single-model forecasts  multi-model is improved compared to poor models  however, multi-model is worse than good models Is there ‘the best model’?? Hagedorn et al. (2005)

8. Antje Weisheimer Meteorological Training Course 27 April 2006 The multi-model ensemble concept: case A DEMETER one-months lead SST anomaly hindcasts (left) and cumulative distribution (right) for JJA 1988 at a single grid point in the tropical Pacific. case A: error cancellation multi-model single-model verification Hagedorn et al. (2005) SST anomalies model ranking cumulative distributions

9. Antje Weisheimer Meteorological Training Course 27 April 2006 The multi-model ensemble concept: case B The identification of ‘the best model’ depends critically on the aspect considered:  variable  region  season  lead time  choice of metric/skill score There is no single best model! rank SST 1987 SST 1988 MSLP 1988

10. Antje Weisheimer Meteorological Training Course 27 April 2006 The multi-model ensemble concept: case C SST anomalies  None of the single-model ensembles predicts the anomaly with Prob ≠ 0  the multi-model ensemble can never be better than the best single model, but will always be better than the worse single-models  Note: multi-model assigns a higher probability to negative anomalies than most single-model ensembles Hagedorn et al. (2005)

11. Antje Weisheimer Meteorological Training Course 27 April 2006  hindcast production period: 1958-2001  9 - member IC ensembles for each model  ERA-40 initial conditions  SST and wind perturbations  4 start dates per year: 1 st of Feb, May, Aug, and Nov  6 month hindcasts The DEMETER project multi-model of 7 coupled general circulation models http://www.ecmwf.int/research/demeter/

12. Antje Weisheimer Meteorological Training Course 27 April 2006 Feb 87 May 87 Aug 87 Nov 87 Feb 88... 7 models x 9 ensemble members  63-member multi-model ensemble = 1 hindcast The DEMETER project Production for 1958-2001 = 44x4 = 176 hindcasts multi-model of 7 coupled general circulation models

13. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: example of Nino3 SST hindcasts Nino3 area ECMWF CNRM UKMO MPI ERA40

14. Antje Weisheimer Meteorological Training Course 27 April 2006 rel. frequency that the verification (ERA-40) lies outside the multi-model ensemble bounding box, based on 6-hourly data DEMETER: capturing the T2m 1989-1998 verification rel. spread of the multi- model ensemble vs. climatology under- over- dispersive systematic errors Weisheimer et al. (2005)

15. Antje Weisheimer Meteorological Training Course 27 April 2006 bounding box insideoutside spread ens/clim <1 >1 DEMETER: capturing the T2m 1989-1998 verification Weisheimer et al. (2005)

16. Antje Weisheimer Meteorological Training Course 27 April 2006 insideoutside <1 >1 1st month DEMETER: capturing the T2m 1989-1998 verification Weisheimer et al. (2005) 2nd month 3rd month capture rate over time days fraction of grid points (in %) start date:

17. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: capturing the T2m 1989-1998 verification rel. frequency that the verification lies out- side the ensemble bounding box multi-model ens single-model ens Weisheimer et al. (2005)

18. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: multi-model vs single-model Relative ACC improvement of the multi-model compared to the single models for JJA from 1980-2001 (one month lead) SST MSLP Anomaly Correlation Coefficients (ACC) multi-model baseline model ranking Hagedorn et al. (2005)

19. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: multi-model vs single-model multi-model baseline model ranking SST MSLP Relative improvement of the multi-model compared to the single models for JJA from 1980-2001 (one month lead) for different scores. Hagedorn et al. (2005) Anomaly Correlation Coefficients (ACC), root mean square skill score (RMSSS), Ranked Probability Skill Score (RPSS) and ROC Skill Score (ROCSS) Tropics

20. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: Brier score of multi-model vs single-model 1959-2001 multi-model single-model Brier skill score Hagedorn et al. (2005)

21. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: Brier score of multi-model vs single-model multi-model single-model Reliability skill score Resolution skill score single-model multi-model Hagedorn et al. (2005)  improved reliability of the multi-model predictions  improved resolution of the multi-model predictions

22. Antje Weisheimer Meteorological Training Course 27 April 2006 BSS Rel-Sc Res-Sc Reliability diagrams (T2m > 0) 1-month lead, start date May, 1980 - 2001 DEMETER: multi-model vs single-model 0.039 0.899 0.141 0.039 0.899 0.140 0.095 0.926 0.169 -0.001 0.877 0.123 0.065 0.918 0.147 -0.064 0.838 0.099 0.047 0.893 0.153 0.204 0.990 0.213 multi- model Hagedorn et al. (2005)

23. Antje Weisheimer Meteorological Training Course 27 April 2006 multi-model minus best single model multi-model minus random chosen single model RPSS, precipitation, 1-month lead, start date November DEMETER: multi-model vs single-model

24. Antje Weisheimer Meteorological Training Course 27 April 2006  Is the multi-model skill improvement due to –increase in ensemble size? –using different sources of information?  An experiment with the ECMWF coupled model and 54 ensemble members to assess –impact of the ensemble size –impact of the number of models DEMETER: impact of ensemble size

25. Antje Weisheimer Meteorological Training Course 27 April 2006 single-model [54 members] multi-model [54 members] 1-month lead, start date May, 1987 - 1999 DEMETER: impact of ensemble size BSS Rel-Sc Res-Sc Reliability diagrams (T2m > 0) 1-month lead, start date May, 1987 - 1999 0.170 0.959 0.211 0.222 0.994 0.227 Hagedorn et al. (2005)

26. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: impact of number of models realisations of different single- model ensembles with the same number of members realisations of different multi- model combinations Hagedorn et al. (2005)

27. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: prediction of tropical storms  GCMs nowadays are able to simulate tropical storms with a seasonal evolution and interannual variability consistent with observations over the western North Atlantic, eastern North Pacific and western North Pacific  frequency of simulated tropical storms is strongly correlated with interannual variability of observed large-scale circulation  operational monthly forecasts of tropical storm frequency at ECMWF (see lecture by Frédéric Vitart)  objective procedure for tropical storms detection tracks low vortices with a warm core above (Vitart et al., 2003)  quality of seasonal prediction of tropical storms may be improved by multi-model combination  DEMETER (Vitart, 2006)

28. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: averaged number of tropical storms 1987-2001 MULTI-MODEL Vitart (2006)

29. Antje Weisheimer Meteorological Training Course 27 April 2006 DEMETER: tropical storms interannual variability 1987-2001 multi-model forecast observations 2  error Vitart (2006) North Atlantic r=0.62 Eastern North Pacific r=0.56 Western North Pacific r=0.72 South Pacific r=0.62

30. Antje Weisheimer Meteorological Training Course 27 April 2006 EUROSIP: European operational Seasonal-to-Interannual Predictions  Three coupled seasonal forecast systems: –ECMWF –Météo France –UK Met Office  All systems are running on ECMWF supercomputers  Hindcast periods –1987-2001 for ECMWF and UK Met Office –1993-2004 for Météo France  Development of multi-model products is ongoing

31. Antje Weisheimer Meteorological Training Course 27 April 2006 observed SST anomalies DJF 2005/2006 ECMWF EUROSIP: European operational seasonal multi-model predictions Courtesy L.Ferranti Météo France UK MetOffice ensemble mean anomalies forecasts started in Nov 2005

32. Antje Weisheimer Meteorological Training Course 27 April 2006 Prob (MSLP<lower tercile) Prob (T2m > upper tercile) EUROSIP: The European winter DJF 2005/2006 Probabilistic multi-model forecasts observed anomalies T2m MSLP

33. Antje Weisheimer Meteorological Training Course 27 April 2006 EUROSIP: The latest forecasts for JJA 2006 EUROSIP forecasts for JJA initialised on April 1 st 2006 Chances for a warm and dry summer are…

34. Antje Weisheimer Meteorological Training Course 27 April 2006 stream 1month 18-24 Three approaches to tackle model uncertainty:  Multi-model: 7 coupled GCMs, each 9 IC ensemble members  Perturbed physics: 2 coupled GCMs, each 9 IC ens. members  Stochastic physics: 1 coupled GCM, 9 ensemble members - hindcast production period: 1991-2001 - seasonal runs (7 months): two start dates per year (May, Nov) - annual runs (14 months): at least one start date per year (Nov) - multi-annual/decadal runs (10 years): starting in 1965 and 1994 - model level data available for 3 of the multi-model GCMs ENSEMBLES: seasonal, interannual and decadal predictions EU funded Integrated Project 09/2004 - 08/2009 http://ensembles-eu.metoffice.com/index.html http://www.ecmwf.int/research/EU_projects/ENSEMBLES/index.html public data dissemination

35. Antje Weisheimer Meteorological Training Course 27 April 2006 ENSEMBLES: seasonal, interannual and decadal predictions SST anomalies in the Nino3 region First decadal hindcast experiments using multi-model, stochastic physics and perturbed parameter ensembles obs ECMWF ECMWF-CASBS (stoch. phys.) GloSea (MetOffice) DePreSyS (pert. phys.)

36. Antje Weisheimer Meteorological Training Course 27 April 2006 Others: IPCC AR4 multi-model climate change simulations 23 coupled state-of-the-art GCMs run with different emission scenarios Weisheimer and Palmer (2005) multi-model histograms Probability of warm European summers in 2081-2100 1971-1990 A2 2081-2100 B1 2081-2100 A1B 2081-2100 A1B+A2+B1 95% central data archive at PCMDI

37. Antje Weisheimer Meteorological Training Course 27 April 2006 Others: ensemble climate simulations with perturbed parameters Quantifying Uncertainty in Model Predictions (QUMP) using a 53-member ensemble based on perturbed physical parameters pdf of climate sensitivity climateprediction.net using a multi-thousand member grand ensemble generated by distributed computing Stainforth et al, 2005 Murphy et al, 2004 temperature distribution

38. Antje Weisheimer Meteorological Training Course 27 April 2006 Summary  The quality of seasonal-to-decadal predictions may be improved by using combined forecasts produced by different models (multi-model ensemble forecasts).  Multi-model ensemble forecasting is a pragmatic and efficient method in filtering out model errors present in the individual ensemble forecasts.  Multi-model predictions yield, on average, more accurate predictions than either of the individual single-model ensembles (e.g., DEMETER).  The improvement is mainly due to more consistency and increased reliability.

39. Antje Weisheimer Meteorological Training Course 27 April 2006 Outlook A. Murphy (1993): What is a good forecast? 1.Consistency: correspondence between forecaster‘s best judgement and their forecasts 2.Quality: correspondence between forecasts and matching observations  Multifaceted nature of forecast evaluation  Measure-orineted and distribution-oriented scores 3.Value: benefits realised by decision makers through the use of the forecasts Paco’s talk after coffee break!

40. Antje Weisheimer Meteorological Training Course 27 April 2006 References (I)  Doblas-Reyes, F.J., M. Déqué and J.-P. Piedelièvre, 2000: Multi-model spread and probabilistic seasonal forecasts in PROVOST. Q.J.R.Meteorol.Soc., 126, 2069-2088.  Doblas-Reyes, F.J., R. Hagedorn and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part II: Calibration and combination. Tellus, 57A, 234-252.  Hagedorn, R., F.J. Doblas-Reyes and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part I: Basic concept. Tellus, 57A, 219-233.  Joliffe, I.T. and D.B. Stephenson (Ed.), 2003: Forecast verification: A practitioner’s guide in atmospheric science. Wiley New York, 240pp.  Murphy, A.H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281-293.  Murphy, J.M. et al, 2004: Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature, 430, 768-772.  Palmer, T.N. et al, 2004: Development of a European multi-model ensemble system for seasonal to inter-annual prediction (DEMETER). Bull. Am. Meteorol. Soc., 85, 853-872.  Stainforth et al, 2005: Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature, 433, 403-406.

41. Antje Weisheimer Meteorological Training Course 27 April 2006 References (II)  Weisheimer, A., L.A. Smith and K. Judd, 2005: A new view of seasonal forecast skill: Bounding boxes from the DEMETER ensemble forecasts. Tellus, 57A, 265-279.  Weisheimer, A. and T.N. Palmer, 2005: Changing frequency of occurrence of extreme seasonal temperatures under global warming. Geophys. Res. Lett., 32, L20721, doi:10.1029/2005GL023365.  Vitart, F., D. Anderson and T. Stockdale, 2003: Seasonal forecasting of tropical cyclone landfall over Mozambique. J. Climate, 16, 3932-3945.  Vitart, F., 2006: Seasonal forecasting of tropical storm frequency using a multi-model ensemble. Q.J.R.Meteorol.Soc., 132, 647-666. Special issue in Tellus (2005), Vol. 57A on DEMETER