Multi-model Short-Range Ensemble Forecasting at Spanish Met Agency (AEMET) J. A. García-Moya, A. Callado, P. Escriba, C. Santos, D. Santos, J. Simarro Spanish Met Service AEMET Training Workshop on NOWCASTING TECHNIQUES Buenos Aires, August 2013

15 August 2013T-Note Workshop2 Outline Introduction EPS for short-range forecast SREPS system at AEMET Verification exercise Against analysis Against synoptic observations Against climate network observations Probabilistic scores Synoptic variables 10 m surface wind speed 6h accumulated precipitation 24h accumulated precipitation Time-Lagged Super-ensemble Comparison with ECMWF EPS Multi-model predictability Conclusions

15 August 2013T-Note Workshop3 Meteorological Framework Main Weather Forecast issues are related with Short-Range forecast of extreme events. Convective precipitation is the most dangerous weather event in the Mediterranean.

15 August 2013T-Note Workshop4 Geographical Framework Western Mediterranean is a close sea rounded by high mountains. In autumn sea is warmer than air. Several cases of more than 200 mm/few hours occurs every year. Some fast cyclogenesis like “tropical cyclones” also appears from time to time (“medicanes”).

15 August 2013T-Note Workshop5 5 Geographical Framework

15 August 2013T-Note Workshop6 6

15 August 2013T-Note Workshop7 7

15 August 2013T-Note Workshop8 Ensemble for Short Range Surface parameters are the most important ones for weather forecast. Forecast of extreme events (convective precip, gales,…) is probabilistic. Short Range Ensemble prediction can help to forecast these events. Forecast risk (Palmer, ECMWF Seminar 2002) is the goal for both Medium- and, also, Short-Range Prediction.

15 August 2013T-Note Workshop9 Errors of short-range forecast Due to model formulation. Due to simplifications in parameterisation schemes. Due to uncertainty in the initial state. Special for LAMs, due to errors in lateral boundary conditions. Due to uncertainties in soil fields (soil temperature and soil water content, …).

15 August 2013T-Note Workshop10 SREPS I Multi-model approach (Hou & Kalnay 2001). Stochastic physics (Buizza et al. 1999). Multi-boundaries: From few global deterministic models. From global model EPS (ECMWF). SLAF technique (Ebisuzaki & Kalnay 1991).

15 August 2013T-Note Workshop11 SREPS II Different assimilation techniques: Optimal Interpolation. Variational (3D or 4D). Perturbed analysis: Singular vectors (ECMWF, Palmer et al. 1997). Breeding (NCEP, Toth & Kalnay 1997). Scaled Lagged Average Forecasting (SLAF, Ebisuzaki & Kalnay 1991). EnKF, ETKF, LETKF, PO

15 August 2013T-Note Workshop12 Multi-model Hirlam ( HRM from DWD (German Weather Service). MM5 ( UM from UKMO (UK Weather Service). LM (COSMO Model) from COSMO consortium ( WRF (NOAA – NCEP) – work in progress

15 August 2013T-Note Workshop13 Multi-Boundaries From different global deterministic models: ECMWF UM from UKMO (UK Weather Service) GFS from NCEP GME from DWD (German Weather Service) CMC from SMC (Canadian Weather Service)

15 August 2013T-Note Workshop14 SREPS at AEMET Mummub: Multi-model Multi-boundaries 72 hours forecast two times a day (00 & 12 UTC). Characteristics: 5 models. 5 boundary conditions. 2 latest ensembles (HH & HH-12). 25 member ensemble every 12 hours Time-lagged Super-Ensemble of 50 members every 12 hours.

15 August 2013T-Note Workshop15

15 August 2013T-Note Workshop16 Post-processing Integration areas 0.25 latxlon, 40 levels Interpolation to a common area ~ North Atlantic + Europe Grid 380x184, 0.25º Software Enhanced PC + Linux ECMWF Metview + Local developments Outputs Deterministic Ensemble probabilistic

15 August 2013T-Note Workshop17 Monitoring in real time Intranet web server Deterministic outputs Models X BCs tables Maps for each couple (model,BCs) Ensemble probabilistic outputs Probability maps: 6h accumulated precipitation, 10m wind speed, 24h 2m temperature trend Ensemble mean & Spread maps EPSgrams (work in progress) Verification:Deterministic & Probabilistic Against ECMWF analysis Against observations

15 August 2013T-Note Workshop18 Monit: all models X bcs Whole Area Zoom over Spain

15 August 2013T-Note Workshop19 Monit: Spread – Ensemble mean maps Spread at key mesoscale areas

15 August 2013T-Note Workshop20 Case Study 06/10/2006 at 00 UTC More than 15 mm/6 hours

15 August 2013T-Note Workshop21 Verification Verification exercise, April-June 2006: Calibration: with synoptic variables Z500, T500, Pmsl Response to binary events: reliability and resolution of surface variables 10m surface wind, 6h and 24h accumulated precipitation

15 August 2013T-Note Workshop22 Interpolation

15 August 2013T-Note Workshop23 Obs verification - Probabilistic scores Ensemble calibration: Synoptic variables: Z500, T500, Pmsl Scores: Rank histograms Spread-skill Response to binary events: Surface variables: 10m surface wind (10,15,20m/s thresholds) 6h accumulated precipitation (1,5,10,20mm thresholds) 24h accumulated precipitation (1,5,10,20mm thresholds) Scores: Reliability, sharpness (H+24, H+48) ROC, Relative Value (H+24, H+48) BSS, ROCA with forecast length

15 August 2013T-Note Workshop24 Summary of Probabilistic Verification

15 August 2013T-Note Workshop25 Synoptic parameters Using ECMWF Analysis as reference: MSLP Over all FC lengths H+00.. H+72: Spread-skill H+72: Rank histograms Using Synoptic observations as reference: MSLP Over all FC lengths H+00.. H+72: Spread-skill H+72: Rank histograms

15 August 2013T-Note Workshop26 MSLP-ECMWF

15 August 2013T-Note Workshop27 MSLP - Obs

15 August 2013T-Note Workshop28 Binary events Binary events X = {0,1} at every point Accumulated precipitation in 24 hours >= 5mm Useful to decompose the forecast in thresholds Performance computed using contingency tables (CT’s)

15 August 2013T-Note Workshop29 Contingency tables It is the best way to characterize a binary event fc(X)={1,0} ob(X)={1,0} ob 10 fc 1aba+b 0cda+d a+cb+da+b+c+d = N

15 August 2013T-Note Workshop30 Contingency tables: scores Several scores can computed from CT’s Base Rates( a + c ) / n HitRate HIRa / ( a + c ) False Alarm Rate FARb / ( b + d ) False Alarm Ratio FARati o b / ( a + b ) Proportion Correct PC( a + d ) / n … ob 10 fc 1aba+b 0cda+d a+cb+da+b+c+d = N

15 August 2013T-Note Workshop31 Example: Every pdf threshold has its own CT ob 10 P(X) >= 0/n 1a0a0 b0b0 0c0c0 d0d0 P(X) >= 1/n 1a1a1 b1b1 0c1c1 d1d1 P(X) >= 2/n 1a2a2 b2b2 0c2c2 d2d2 … P(X) >= n/n 1anan bnbn 0cncn dndn HIR 0 FAR 0 HIR 1 FAR 1 HIR 2 FAR 2 HIR n FAR n

15 August 2013T-Note Workshop32 Reliability Observation frequency conditioned to forecast probability Reliability diagrams ( i/N, Δa i /(Δa i +Δb i ) ) i = 0…N Near diagonal ~ reliable Sharpness histograms ( i/N, Δa i +Δb i ) i = 0…N U shape ~ sharp (discriminating binary event)

15 August 2013T-Note Workshop33 Precipitation Using ECMWF Deterministic Model as reference: 6 hours accumulation – 24 hours forecast length 24 hours accumulation – 54 hours forecast length Thresholds 1, 5, 10 y 20 mm Using Synoptic observations as reference: 6 hours accumulation – 24 hours forecast length 24 hours accumulation – 54 hours forecast length Thresholds 1, 5, 10 y 20 mm

15 August 2013T-Note Workshop34 ECMWF Reliab h Acc. Precip H+54 (1,5,10,20) mm Reliab. - 6 h Acc. Precip H+24 (1,5,10,20) mm

15 August 2013T-Note Workshop35 Observations Reliab. - 6 h Acc. Precip H+24 (1,5,10,20) mm Reliab h Acc. Precip H+54 (1,5,10,20) mm

15 August 2013T-Note Workshop36 Resolution Based on Signal Detection Theory ROC (Relative Operating Characteristics) ( FAR i, HIR i ) i = 0…N ROCArea ~ Resolution or binary event discrimination Forecast conditioned by observation

15 August 2013T-Note Workshop37 ROC curves – 24 h Acc Precip (1, 5, 10 & 20 mm) ECMWF Observations

15 August 2013T-Note Workshop38 ECMWF - Analysis ROC 10 m wind H+24 (10,15,20) m/s Reliability

15 August 2013T-Note Workshop39 Observations 10 m wind H+24 (10,15,20) m/s Reliability ROC

15 August 2013T-Note Workshop40 Joint

15 August 2013T-Note Workshop41 Reliability & Sharpness Good reliability according to thresholds (base rate) forecast length No Under- sampling

15 August 2013T-Note Workshop42 Resolution Good resolution ROC Areas BSSs Good RV curves

15 August 2013T-Note Workshop43 Time-Lagged Super-ensemble How much predictability can be added by a time-lagged super-ensemble? 40 members super-ensemble (SE-SREPS) with the last two runs of SREPS ( HH & HH-12). Verifications against observations Cheap in terms of computer resources Just a different post-process

15 August 2013T-Note Workshop44 Spread-skill

15 August 2013T-Note Workshop45 Rank histogram

15 August 2013T-Note Workshop46 Reliability diagrams

15 August 2013T-Note Workshop47 Comparison with ECMWF EPS Period: April to June 2006 European Synop obs: H+72. Mslp / v 10m / Precipitation European climate precipitation network: H+54 (longest SREPS period matching observations). 24 hours accumulated precipitation (from early morning to early morning).

15 August 2013T-Note Workshop48 Climate Obs – 24 h. Precip (1, 5, 10 & 20 mm)

15 August 2013T-Note Workshop49 Climate Obs – 24 h. Precip (1, 5, 10 & 20 mm)

15 August 2013T-Note Workshop50 Climate Obs – 24 h. Precip (1, 5, 10 & 20 mm) Brier Skill Score (BSS) Decomposition – 10 6 realizations Precip (mm)Climatolog. Frequency UncertaintyMumMubECMWF 21ECWMF Precip (mm)MumMubECMWF 21ECWMF Precip (mm)MumMubECMWF 21ECWMF

15 August 2013T-Note Workshop51 24 hours accumulated precipitation from European climate network upscaled to 0.25 deg. Resolution - BSS

15 August 2013T-Note Workshop52 Why Multi-model? Better representation of model errors (SAMEX, Hou & Kalnay 2001, and DEMETER). Consistent set of perturbations of initial state and boundaries. Better results than any single model ensemble (SAMEX, DEMETER, Arribas et al., 2005).

15 August 2013T-Note Workshop53 Single model Ensembles (4 members each)

15 August 2013T-Note Workshop54 BMA is a statistical method for postprocessing ensembles based on standard method for combining predictive distributions from different sources. The BMA predictive PDF of any quantity of interest is a weighted average of PDFs centred on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models’ relative contributions to predictive skill over the training period. Raftery et al, 2005 BMA INTRO

15 August 2013T-Note Workshop55 BMA FUNDAMENTALS F2F2 F1F1 F4F4 F5F5 F3F3 BMA predictive PDF

15 August 2013T-Note Workshop56 BMA calibration using TEMP and SYNOP observations over whole area : 500 hPa Temperature (T500) 500 hPa Geopotencial (Z500) 10m Wind speed (S10m) 3, 5 and 10 days of training period 3 months of calibration (April, May and June of 2006) for Z500 and T500 and 1 month for S10m (April 2006) 24, 48 and 72 hours forecast for Z500 and T500 and 24 and 72 hours forecast for S10m BMA EXPERIMENTS

15 August 2013T-Note Workshop57 RESULTS MULTIMODEL BMA 3 T. DAYS BMA 5 T. DAYS BMA 10 T. DAYS

15 August 2013T-Note Workshop58 RESULTS MULTIMODEL BMA 3 T. DAYS BMA 5 T. DAYS BMA 10 T. DAYS

15 August 2013T-Note Workshop59 RESULTS MULTIMODEL BMA 3 T. DAYS BMA 5 T. DAYS BMA 10 T. DAYS

15 August 2013T-Note Workshop60 WEIGHT TIME SERIES HRM HIRLAM LOKAL MODEL MM5 UM

15 August 2013T-Note Workshop61 WEIGHT TIME SERIES HIRLAM

15 August 2013T-Note Workshop62 WEIGHT TIME SERIES HIRLAM

15 August 2013T-Note Workshop63 WEIGHT TIME SERIES HIRLAM

15 August 2013T-Note Workshop64 Summary I A Multi-model-Multi-boundaries Short Range Ensemble Prediction System (MMSREPS), has been developed at the AEMET-Spain We show here 3months verification results (April-June 21006), against both synoptic and climate observations and ECMWF analysis and model: Response to binary events: reliability and resolution of surface variables 10m surface wind, 6h and 24h accumulated precipitation These results look promising: Verification against ECMWF analysis and model shows very good results Verification against observations shows quite good results Ensemble is a bit under-dispersive Good response to binary events (synoptic and climate observations)

15 August 2013T-Note Workshop65 Summary II A Time-Lagged Super-ensemble with 40 members had shown better performance than the Multi- model SREPS alone. Multi-model technique gives much better spread than any other single-model technique Bias correction and calibration through a Bayesian Model Averaging scheme is under development (first results show better performance).

15 August 2013T-Note Workshop66 Questions