Presentation is loading. Please wait.

Presentation is loading. Please wait.

Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF.

Similar presentations


Presentation on theme: "Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF."— Presentation transcript:

1 Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF

2 Slide 2ECMWF forecast User Meeting -- Reading, June 2006 Overview Deterministic forecast performance for different weather parameters Precipitation forecast: scores and their confidence SYNOP on the GTS Precipitation analysis Ensemble prediction System: its performance relative to precipitation and 10m wind

3 Slide 3ECMWF forecast User Meeting -- Reading, June 2006 North America Europe 2m Temperature Skill (rmse) for different forecast ranges Top panel: North America Bottom panel: Europe Strong seasonality in the skill with higher values during winters. 2mT skill have reached a plateau. The skills for different timesteps have levelled off. The two past winters were dominated by prolonged periods of cold weather.

4 Slide 4ECMWF forecast User Meeting -- Reading, June 2006 Performance for different forecast ranges Europe Top panel: Specific humidity Bottom panel: 10m wind speed Higher skill (MAE) in winter. Last four winters have consistently kept higher level of performance RMSE: Changes in the forecasting model have improved the performance of the model in forecasting wind speed.

5 Slide 5ECMWF forecast User Meeting -- Reading, June 2006 Observed yesObserved no Forecast yes Forecast no 1. FREQUENCY BIAS INDEX 2. TRUE SKILL SCORE 3. HIT RATE 3. FALSE ALARM RATE

6 Slide 6ECMWF forecast User Meeting -- Reading, June mm/24h Europe 24 hour accumulated precipitation verified against SYNOP on GTS The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value. High thresholds have large confidence intervals, important to remember when assessing performance of the system t+42 t+90

7 Slide 7ECMWF forecast User Meeting -- Reading, June 2006 Europe 24 hour accumulated precipitation verified against SYNOP on GTS The forecasting system over- estimate the number of events for thresholds of 1 mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented. Further changes in the convection during 2003 have slightly increased FBI values FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimate FBI<1 under-estimate t+42: solid shading t+90: dotted shading

8 Slide 8ECMWF forecast User Meeting -- Reading, June 2006 FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals! 24 hour accumulated precipitation verified against SYNOP on GTS Europe 15mm/24h t+42: solid shading t+90: dotted shading

9 Slide 9ECMWF forecast User Meeting -- Reading, June mm/24h DJF MAM Green: t+42 Light blue: t+90 15mm/24h Stratifying the sample: selecting seasons as rule

10 Slide 10ECMWF forecast User Meeting -- Reading, June 2006 Precipitation analysis for Europe High density networks in Europe (Member and Co-operating states) Upscaling (simple box averaging to obtain a areal precipitation value)

11 Slide 11ECMWF forecast User Meeting -- Reading, June 2006 Each grid box will contain a certain number of stations. The number of stations will not be constant every day. The number of stations per grid box indicates how representative the analysis is for the specific grid point.

12 Slide 12ECMWF forecast User Meeting -- Reading, June 2006 Green : proxy t+42 t+90 1 mm/24h Which reference system? Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.

13 Slide 13ECMWF forecast User Meeting -- Reading, June mm/24h t+90 Which reference system? Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges. Green : proxy

14 Slide 14ECMWF forecast User Meeting -- Reading, June 2006 Europe FBI plotted for two thresholds (1mm/24h, and 15mm/24h) Verification against SYNOP on the GTS (greenish shading), Verification against precipitation analysis (blue dotted) FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds. Forecast range t+66

15 Slide 15ECMWF forecast User Meeting -- Reading, June 2006 Europe TSS (threshold 25mm/24h) plotted for t+66 Verification against SYNOP on the GTS (greenish shading), Verification against precipitation analysis (blue dotted) TSS values decrease as we increase forecast range.

16 Slide 16ECMWF forecast User Meeting -- Reading, June 2006 Timeseries of Brier Skill Score for Europe The BSS is written as 1- BS/BS ref Sample climate is the reference system BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast Forecast vs. observations Improvements back in Autumn 1999 – High thresholds performance down at the beginning of 2005 probably linked to drier conditions over Europe. Dec-Apr 2006 performance better than previous 3 years. Increased resolution T255 T399

17 Slide 17ECMWF forecast User Meeting -- Reading, June 2006 Forecast vs. observations

18 Slide 18ECMWF forecast User Meeting -- Reading, June 2006 Timeseries of Brier Skill Score for Europe The BSS is written as 1- BS/BS ref Sample climate is the reference system BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C Forecast vs analysis Increased resolution T255T m Wind speed

19 Slide 19ECMWF forecast User Meeting -- Reading, June 2006 Europe Rainy season (24h accum.): October to April Forecast range: t+96 Verification against SYNOP on GTS mm/24h BS= mm/24h BS=0.157 Consistent picture for the two seasons

20 Slide 20ECMWF forecast User Meeting -- Reading, June 2006 Europe Rainy season (24h accum.): October to April Forecast range: t+96 Verification against SYNOP on GTS mm/24h BS= mm/24h BS=0.019 Consistent picture for the two seasons. Higher thresholds better reliability

21 Slide 21ECMWF forecast User Meeting -- Reading, June 2006 Europe Rainy season (24h accum.): October to April Forecast range: t+96 Verification against SYNOP on GTS mm/24h mm/24h Consistent picture for the two seasons. Full symbol: T511/799 Shape: T255/T399 t+96 : A=0.864 t+120: A=0.832 t+96 : A=0.889 t+120: A=0.860

22 Slide 22ECMWF forecast User Meeting -- Reading, June 2006 Increased resolution Europe ROC Area Verification against SYNOP on GTS for t+96 T255 T399

23 Slide 23ECMWF forecast User Meeting -- Reading, June 2006 Conclusion 2m Temperature: the skill over Europe has reached a plateau. Over N.America winters are more skilful than summers. Specific humidity shows consistent skills. Winters more skilful than summers Wind: Recent changes in the model have resulted in a decrease of the RMSE over Europe. TCC: New cloud scheme was introduced in April Forecasts of certain cloud types have improved. Importance of confidence intervals Precipitation forecast improvements are slow, but evident. FBI indicates over-estimation of small threshold events verification against precipitation analysis shows a better picture. Precipitation analysis can be used for verification in a delayed mode. It can be used to define the performance of a reference system.

24 Slide 24ECMWF forecast User Meeting -- Reading, June 2006 Brier skill score and ROC area: In recent year the system has maintained its good performance. The increase in horizontal resolution has had a small but positive impact on both precipitation and wind speed. Reliability diagrams for the last two rainy seasons show a skilful system, with an evident decrease of Brier Score for the season. Conclusion


Download ppt "Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF."

Similar presentations


Ads by Google