Slide 1ECMWF forecast products users meeting – Reading, June 2005 Verification of weather parameters Anna Ghelli, ECMWF

Slide 2ECMWF forecast products users meeting – Reading, June 2005 Overview Deterministic forecast performance for different weather parameters Precipitation forecast: scores and their confidence SYNOP on the GTS Precipitation analysis Ensemble prediction System: its performance relative to precipitation

Slide 3ECMWF forecast products users meeting – Reading, June 2005 North America Europe 2m Temperature Skill (rmse) for different forecast ranges Top panel: North America Bottom panel: Europe Higher skill in winter. The positive trend of the timeseries for both winter and summer periods indicates continuous forecast improvements. Higher skill in winter, interrupted in 2005 by a period of strong inversion at low levels in Central Europe which has not been represented properly by the model..

Slide 4ECMWF forecast products users meeting – Reading, June 2005 Europe Skill for different forecast ranges Top panel: Specific humidity Bottom panel: 10m wind speed Higher skill (MAE) in winter. Last four winters have consistently kept higher level of performance Skill (RMSE): Changes in the forecasting model have not greatly improved the performance of the model in forecasting wind speed.

Slide 5ECMWF forecast products users meeting – Reading, June 2005 The skill has been averaged over a year Skill (rmse) plotted vs forecast day for different parameters. Total Cloud Cover: black 2m temperature: blue

Slide 6ECMWF forecast products users meeting – Reading, June 2005 Observed yesObserved no Forecast yes Forecast no 1. FREQUENCY BIAS INDEX 2. TRUE SKILL SCORE 3. HIT RATE 3. FALSE ALARM RATE

Slide 7ECMWF forecast products users meeting – Reading, June mm/24h 25mm/24h Europe 24 hour accumulated precipitation verified against SYNOP on GTS The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value. High thresholds have large confidence intervals, important to remember when assessing performance of the system t+42 t+66

Slide 8ECMWF forecast products users meeting – Reading, June 2005 Europe 24 hour accumulated precipitation verified against SYNOP on GTS The forecasting system over- estimate the number of events for thresholds of 1mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented. FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimate FBI<1 under-estimate t+42: solid shading t+66: dotted shading

Slide 9ECMWF forecast products users meeting – Reading, June 2005 FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals! 24 hour accumulated precipitation verified against SYNOP on GTS Europe 5mm/24h t+42: solid shading t+66: dotted shading

Slide 10ECMWF forecast products users meeting – Reading, June 2005 Precipitation analysis for Europe High density networks in Europe (Member and Co-operating states) Upscaling (simple box averaging to obtain a areal precipitation value)

Slide 11ECMWF forecast products users meeting – Reading, June 2005 Each grid box will contain a certain number of stations. The number of stations will not be constant every day. The number of stations per grid box indicates how representative the analysis is for the specific grid point.

Slide 12ECMWF forecast products users meeting – Reading, June 2005 Europe FBI plotted for two thresholds (0.25mm/24h, and 10mm/24h) Verification against precipitation analysis (yellow shading), Verification against SYNOP on GTS (blue dotted) FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds. 0.25mm/24h 10mm/24h Forecast range t+42

Slide 13ECMWF forecast products users meeting – Reading, June 2005 Europe TSS (threshold 0.25mm/24h) plotted for two forecast ranges: t+42 (top) and t+90 (bottom) Verification against precipitation analysis (yellow shading), Verification against SYNOP on GTS (blue dotted) TSS values decrease as we increase forecast range. In January 2003 there was a model change: improved cloud scheme numerics, revised cloud scheme and convection 0.25mm/24h t+42 t+90

Slide 14ECMWF forecast products users meeting – Reading, June 2005 Europe TSS plotted for two thresholds (5mm/24h, and 15mm/24h) Verification against precipitation analysis (yellow shading), Verification against SYNOP on GTS (blue dotted) TSS values are higher for winter months. Confidence intervals become larger as threshold increases. Forecast range t+90 5mm/24h 15mm/24h

Slide 15ECMWF forecast products users meeting – Reading, June 2005 Timeseries of Brier Skill Score for Europe The BSS is written as 1- BS/BS ref Sample climate is the reference system BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast Forecast vs. observations Improvements back in Autumn 1999 – High thresholds performance down at the beginning of 2005 linked to drier conditions over Europe? Increased resolution

Slide 16ECMWF forecast products users meeting – Reading, June 2005 Timeseries of Brier Skill Score for Europe The BSS is written as 1- BS/BS ref Sample climate is the reference system BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C Forecast vs proxy Increased resolution

Slide 17ECMWF forecast products users meeting – Reading, June 2005 Europe Rainy season: October to April Forecast range: t+96 Verification against SYNOP on GTS mm/24h BS= mm/24h BS=0.157 Consistent picture for the two seasons

Slide 18ECMWF forecast products users meeting – Reading, June 2005 Europe Rainy season: October to April Forecast range: t+96 Verification against SYNOP on GTS mm/24h BS= mm/24h BS=0.045 Consistent picture for the two seasons. Higher thresholds better reliability

Slide 19ECMWF forecast products users meeting – Reading, June 2005 Europe Rainy season: October to April Forecast range: t+96 Verification against SYNOP on GTS mm/24h mm/24h Consistent picture for the two seasons. Full symbol: T511 Shape: T255

Slide 20ECMWF forecast products users meeting – Reading, June 2005 Increased resolution Europe ROC Area Verification against SYNOP on GTS for t+96 Drier conditions over Europe?

Slide 21ECMWF forecast products users meeting – Reading, June 2005 Conclusion 2m Temperature: positive trends show increased skills for Europe and North America. Strong inversion in the winter was not properly forecast by the T511. Specific humidity shows increased skills. Winters more skilful than summers Wind: The changes in the model have not brought large improvements in the wind speed forecast TCC: small improvements in forecast skill. New cloud scheme was introduced in April 2005 Importance of confidence intervals Precipitation forecast improvements are slow, but evident. FBI indicates over-estimation of small threshold events verification against precipitation analysis shows a better picture. Precipitation analysis can be used for verification in a delayed mode. The number of station per grid box gives and indication on how representative is the analysis at any grid point

Slide 22ECMWF forecast products users meeting – Reading, June 2005 Brier skill score and ROC area: increase in resolution has improved the system. In recent year the system has maintained its good performance. The drier conditions of the recent winter show up in the timeseries small sample size effects? Reliability diagrams for the last two rainy seasons show a skilful system Conclusion