Presentation is loading. Please wait.

Presentation is loading. Please wait.

Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation.

Published byDebra Cain Modified over 8 years ago

Similar presentations

Presentation on theme: "Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation."— Presentation transcript:

1 Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation COSMO General Meeting, Offenbach, 07.-11.09.2009 Tanja Weusthoff

2 2 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Fuzzy Verification „multi-scale, multi-intensity approach“ „Fuzzy verification toolbox“ of B. Ebert Two methods Upscaling (UP) Fraction Skill Score (FSS) present output scale dependent standard setting: 3h accumulations COSMO-2 (2.2km): leadtimes 03-06 COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15 fcstobs increasing box size increasing threshold

3 3 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff … Upscaling (UP) 1. Principle: Define box around region of interest and calculate the average of observation and forecast data within this box. 3. Equitable Threat Score (ETS) R ave Event if R ave ≥ threshold No-Event if R ave < threshold yesno yesHit False Alarm noMiss Correct negative observation forecast 2. Contingency Table Q: Which fraction of observed yes - events was correctly forecast? (Atger, 2001)

4 4 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) XX XX XX xX X X x 1. Principle: Define box around region of interest and determine the fraction p j and o j of grid points with rain rates above a given threshold. 3. Skill Score for Probabilities 2. Probabilities Q: On which spatial scales does the forecast resemble the observation? FBS worst  no colocation of non-zero fractions 0 threshold

5 5 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) 4. Useful Scales useful scales are marked in bold in the graphics

6 6 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Fuzzy Verification - Methods Upscaling - ETSFraction Skill Score - FSS Roberts & Lean (2005) Score: FSS for fractions / probabilities (0 = mismatch, 1 = perfect match) Score: ETS (Equitable Threat Score) (-1/3 = mismatch, 1 = perfect match) „Spatial averaging damps out areas with high rain rates and spreads areas of lower rain rates.“ (B.Ebert,2009) „ETS tends to increase with white space (rare events).“ (B.Ebert,2009) „The absolute value of the FSS is less useful than the scale where an acceptable level of skill is reached.“ (Mittermaier,2008) “The score is most sensitive to rare events.” (Roberts, 2008)

7 7 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff good bad COSMO-7 better COSMO-2 better COSMO-2COSMO-7Difference - = - = Upscaling and Fractions Skill Score Jun – Nov 2007 Fractions skill score Upscaling

8 8 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff COSMO-2 vs. COSMO-7 DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better abs(median) / 0.5(q95-q05) [ 10, Inf ] [ 2, 5 [ [ 5, 10 [ [ 1, 2 [ [ 0, 1 [ Values = Score of COSMO-2 Size of numbers = abs(Median) / 0.5(q95-q05)  measure for significance of differences ¦ COSMO-2 – COSMO-7 ¦ q(50%) q(95%) q(5%) 5% 95%

9 9 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Wilcoxon ranked probability test???

10 10 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Sensitivities Only 00 and 12 UTC model runs COSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15  absolute values of COSMO-2 slightly lower, but still the same pattern of differences DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

11 11 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Weather type verification: COSMO-2 4755745312315 347 # cases Threshold [mm/3h] Upscaling Fractions Skill Score Window size: 3gp, 6.6 kmWindow size: 27gp, 60 km Window size: 3gp, 6.6 km Window size: 27gp, 60 km

12 dx0.10.20.51.02.05.01020 NE45 ///// N ////// NW339927 // SE11119/// S111119// SW1139945// E91527////45 W113915/// F 27 45/// H27 45//// L39152745/// dx0.10.20.51.02.05.01020 NE15/////// N//////// NW3355915// SE11139/// S111339// SW33599/// E15/////// W33359/// F99 //// H/ ////// L3599 /// COSMO-2, gridpoints* 2.2 kmCOSMO-7, gridpoints* 6.6 km Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition. + - + - 7 4 23 5 7 45 5 31 15 34 7 # cases % obs gridpts >= thresh (whole period) 1614107520.5<0.11614107520.4<0.1

13 13 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff %0.10.20.51.02.05.01020 NE764210.3<0.1 N210.80.40.1<0.1 0 NW13118520.70.1<0.1 SE1715129620.4<0.1 S21191511830.70.1 SW252218139310.2 E3210.70.2<0.1 W1815118410.2<0.1 F1715118520.80.2 H220.80.50.20.1<0.1 L423627191130.80.1 ALL1614107520.5<0.1 Fraction of observation gridpoints >= threshold  climatology

14 14 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Frequency of Weather Classes, June – November 2007 4 7 55 7 7 45 31 23 15 34

15 15 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Northerly Winds (NE,N) 11 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in northerly wind situations clearly worse than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

16 16 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Southerly Winds (SE,S) 12 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in southerly wind situations clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

17 17 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Northwesterly winds (NW) 23 days COSMO-2 vs.COSMO-7 COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

18 18 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Flat (F) 15 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all) DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better  COSMO-2 in flat pressure situations clearly worse than over whole period.

19 19 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Summary / Conlusions Fuzzy verification of the D-PHASE operations period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences the results for different weather types show large variations best results were found for southerly winds and winds from Northwest and West, Northeasterly winds as well as Flat pressure situations lead to worse perfomance of both models

20 20 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff UP with other scores POFD = FA/(CN+FA) Probability of false detection (Perfect score: 0) FAR = FA/(H+FA) False Alarm Ratio (Perfect score: 0) POD = H/(H+M) Probability of Detection (Perfect score: 1) BIAS = (H+FA) / (H+M) Frequency Bias (perfect score: 1) HK =POD – FAR True Skill Statistik (perfect score: 1) OR = (H*CN) / (M*FA) Odds Ratio (perfect score: infinity)

21 21 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff dx0.10.20.51.02.05.01020 NE N NW SE S SW E W F H L dx0.10.20.51.02.05.01020 NE N NW SE S SW E W F H L COSMO-2, gridpoints* 2.2 kmCOSMO-7, gridpoints* 6.6 km UP „useful scales“? How to define usefuls scales for ETS?

22 22 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Fuzzy Verifikation Intensity Scale (IS) (Casati et al. 2004) Transformation of Fcst and Obs into binary images on a rain/no-rain basis for the rainfall rate thresholds. Difference between forecast and observation = binary error image. Decomposition into the sum of components at different spatial scales by performing a two dimensional discrete wavelet decomposition. Binary forecastBinary observation Score: Mean squared error (MSE) and MSE skill score (SS) for each spatial scale component of the binary error image. What is the relative improvement of the forecast over some reference forecast?

23 23 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Intensity Scale COSMO-2COSMO-7

24 24 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Intensity Scale COSMO-2

25 25 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Outlook operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score season, year weather-type dependant Intensity scale results will further be investigated (new developments of B. Casati)

26 26 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff All Weatherclasses Overview over all weatherclasses – differences COSMO-2 minus COSMO-7 and absolute values of COSMO-2, here without bootstrapping

27 27 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Northerly Winds (NE,N,NW) NNW NE 23 days 4 days7 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

28 28 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Southerly Winds (SE,S,SW) S SE SW 7 days 45 days 5 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

29 29 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff East and West (E,W) EW 31 days5 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

30 30 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Flat, High, Low (F,H,L) FHL 34 days7 days15 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

31 31 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff Summary JJASONDOP Useful scales of COSMO-2 are the same for all three time periods considered: <= 0.2 mm/3h  scales larger or equal 2.2 km (= 1* gridscale ) 0.5 mm/3h  scales larger or equal 6.6 km (= 3* gridscale ) 1.0 mm/3h  scales larger or equal 19.8 km (= 9* gridscale ) 2.0 mm/3h  scales larger or equal 33.0 km (= 15* gridscale ) 5.0 mm/3h  scales larger or equal 99.0 km (= 45* gridscale ) Most prominent advantage for COSMO- 2! Differences are significant. COSMO-2 better but on many scales only slightly. Advantages on small scales/all thresholds and large thresholds/large scales. Differences are significant. COSMO-2 better than COSMO-7 on all scales. Differences are significant. Most frequent weatherclasses: SW (32 days) and W (21 days). Most frequent weatherclasses: H (24 days) and NW (17 days). Most frequent weatherclasses: SW (45 days) and H (34 days).

32 32 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff JJASONDOP NE -Relatively low values for UP -FSS: COSMO-2 better for thresholds <= 2 mm/3h, useful scales only for largest window size (99 km) - COSMO-2 better on nearly all scales, especially for thresholds of 2-5 mm/3h. Large threshold gave no result in UP  only small - scale events (averaging damps values out). Apparently, COSMO-2 could simulate those events quite well. -Low absolute values (FSS < 0.6, UP < 0.3), no useful scales -COSMO-2 clearly better for thresholds <= 2 mm/3h (FSS) - UP: the averaged precipitation was equally well represented in both models. Only 2 mm/3h threshold better captured in COSMO-2. N Only few days – no clear superiority of one model. Absolute values relatively low. No useful scales. NW - COSMO-2 performed better than COSMO-7, especiall for large thresholds (> 1 mm/3h, FSS and UP). -Many days with that weather type -Over most scales rarely different behaviour -Thresholds 5-10 mm/3h better in COSMO- 2 as well as spatial scale of 6.6 km - COSMO-2 performed better than COSMO-7, especiall for large thresholds (>= 5 mm/3h, FSS and UP). Summary Weatherclasses I

33 33 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff JJASONDOP SE -Only few days; similar behaviour in all time periods: COSMO-2 better, especially for thresholds around 1 mm/3h and small spatial scales (FSS) and large spatial scales (UP) -Large absolute values (e.g. FSS up to 0.9, UP up to 0.66 for DOP) S - Only few days; COSMO-2 better on most scales, most clearly for low thresholds concerning averaging (UP) and large thresholds and small spatial scales regarding probabilities (FSS), in JJA COSMO-2 clearly better also for smallest eindow size (6.6 km) for all thresholds SW - Clearly better performance of COSMO-2, especially on smaller spatial scales - Similar performance of both models, but COSMO-2 better for moderate thresholds (1-2 mm/3h) - Clearly better performance of COSMO-2, especially on smaller spatial scales Summary Weatherclasses II

34 34 Weather type dependant fuzzy verification | COSMO GM, 07.-11.09.2009 Tanja Weusthoff JJASONDOP E -Only few cases (5 for DOP), COSMO-2 clearly better for low thresholds (<= 1 mm), while for a threshold of 2 mm/3h COSMO-7 better on largest spatial scales -Rather low absolute values W - COSMO-2 clearly better on all scales, especially for thresholds <= 2 mm and smaller spatial scales - COSMO-2 slightly better only on small spatial scales - COSMO-2 better for thresholds <= 2 mm and especially smaller spatial scales F -Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds. -Comparable low absolute values, rarely difference in the performance of the two models. -COSMO-2 slightly better for small thresholds - Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds. H -very low absolute values (especially in SON), but COSMO-2 clearly better than COSMO-7 for precipitation thresholds <= 2 mm/3h L - Only few cases (7 for DOP), COSMO-2 better for small spatial scales and thresholds <= 2 mm/3h Summary Weatherclasses III

Download ppt "Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation."

Similar presentations

Robin Hogan Ewan OConnor University of Reading, UK What is the half-life of a cloud forecast?

Robin Hogan Ewan OConnor University of Reading, UK What is the half-life of a cloud forecast?
Validation of Satellite Precipitation Estimates for Weather and Hydrological Applications Beth Ebert BMRC, Melbourne, Australia 3 rd IPWG Workshop / 3.

Validation of Satellite Precipitation Estimates for Weather and Hydrological Applications Beth Ebert BMRC, Melbourne, Australia 3 rd IPWG Workshop / 3.
Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Institut für Physik der Atmosphäre On the Value of.

Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Institut für Physik der Atmosphäre On the Value of.
Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Quantitative precipitation forecasts in the Alps – first.

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Quantitative precipitation forecasts in the Alps – first.
Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz New automatic weather type classifications at MeteoSwiss.

Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz New automatic weather type classifications at MeteoSwiss.
COSMO General Meeting Zurich, 2005 Institute of Meteorology and Water Management Warsaw, Poland- 1 - Verification of the LM at IMGW Katarzyna Starosta,

COSMO General Meeting Zurich, 2005 Institute of Meteorology and Water Management Warsaw, Poland- 1 - Verification of the LM at IMGW Katarzyna Starosta,
WWOSC 2014 Assimilation of 3D radar reflectivity with an Ensemble Kalman Filter on a convection-permitting scale WWOSC 2014 Theresa Bick 1,2,* Silke Trömel.

WWOSC 2014 Assimilation of 3D radar reflectivity with an Ensemble Kalman Filter on a convection-permitting scale WWOSC 2014 Theresa Bick 1,2,* Silke Trömel.
COSMO General Meeting – Moscow 06-10 Sept 2010 Some results from operational verification in Italy Angela Celozzi - Federico Grazzini Massimo Milelli -

COSMO General Meeting – Moscow Sept 2010 Some results from operational verification in Italy Angela Celozzi - Federico Grazzini Massimo Milelli -
1 On the use of radar data to verify mesoscale model precipitation forecasts Martin Goeber and Sean Milton Model Diagnostics and Validation group Numerical.

1 On the use of radar data to verify mesoscale model precipitation forecasts Martin Goeber and Sean Milton Model Diagnostics and Validation group Numerical.
4th Int'l Verification Methods Workshop, Helsinki, 4-6 June 2009 1 Methods for verifying spatial forecasts Beth Ebert Centre for Australian Weather and.

4th Int'l Verification Methods Workshop, Helsinki, 4-6 June Methods for verifying spatial forecasts Beth Ebert Centre for Australian Weather and.
4IWVM - Tutorial Session - June 2009 Verification of categorical predictands Anna Ghelli ECMWF.

4IWVM - Tutorial Session - June 2009 Verification of categorical predictands Anna Ghelli ECMWF.
Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Statistical Characteristics of High- Resolution COSMO.

Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Statistical Characteristics of High- Resolution COSMO.
How can LAMEPS * help you to make a better forecast for extreme weather Henrik Feddersen, DMI * LAMEPS =Limited-Area Model Ensemble Prediction.

How can LAMEPS * help you to make a better forecast for extreme weather Henrik Feddersen, DMI * LAMEPS =Limited-Area Model Ensemble Prediction.
SEASONAL COMMON PLOT SCORES 2012-2013 A DRIANO R ASPANTI P ERFORMANCE DIAGRAM BY M.S T ESINI Sibiu - Cosmo General Meeting 2-5 September 2013.

SEASONAL COMMON PLOT SCORES A DRIANO R ASPANTI P ERFORMANCE DIAGRAM BY M.S T ESINI Sibiu - Cosmo General Meeting 2-5 September 2013.
© Crown copyright Met Office Preliminary results using the Fractions Skill Score: SP2005 and fake cases Marion Mittermaier and Nigel Roberts.

© Crown copyright Met Office Preliminary results using the Fractions Skill Score: SP2005 and fake cases Marion Mittermaier and Nigel Roberts.
We carried out the QPF verification of the three model versions (COSMO-I7, COSMO-7, COSMO-EU) with the following specifications: From January 2006 till.

We carried out the QPF verification of the three model versions (COSMO-I7, COSMO-7, COSMO-EU) with the following specifications: From January 2006 till.
On the spatial verification of FROST-2014 precipitation forecast fields Anatoly Muraviev (1), Anastasia Bundel (1), Dmitry Kiktev (1), Nikolay Bocharnikov.

On the spatial verification of FROST-2014 precipitation forecast fields Anatoly Muraviev (1), Anastasia Bundel (1), Dmitry Kiktev (1), Nikolay Bocharnikov.
Model validation Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015.

Model validation Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015.
Latest results in verification over Poland Katarzyna Starosta, Joanna Linkowska Institute of Meteorology and Water Management, Warsaw 9th COSMO General.

Latest results in verification over Poland Katarzyna Starosta, Joanna Linkowska Institute of Meteorology and Water Management, Warsaw 9th COSMO General.
Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Priority project « Advanced interpretation and verification.

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Priority project « Advanced interpretation and verification.

Similar presentations

Ads by Google