Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz Weather type dependant fuzzy verification of precipitation COSMO General Meeting, Offenbach, Tanja Weusthoff

2 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Fuzzy Verification „multi-scale, multi-intensity approach“ „Fuzzy verification toolbox“ of B. Ebert Two methods Upscaling (UP) Fraction Skill Score (FSS) present output scale dependent standard setting: 3h accumulations COSMO-2 (2.2km): leadtimes COSMO-7 (6.6km): leadtimes 03-06,06-09,09-12,12-15 fcstobs increasing box size increasing threshold

3 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff … Upscaling (UP) 1. Principle: Define box around region of interest and calculate the average of observation and forecast data within this box. 3. Equitable Threat Score (ETS) R ave Event if R ave ≥ threshold No-Event if R ave < threshold yesno yesHit False Alarm noMiss Correct negative observation forecast 2. Contingency Table Q: Which fraction of observed yes - events was correctly forecast? (Atger, 2001)

4 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) XX XX XX xX X X x 1. Principle: Define box around region of interest and determine the fraction p j and o j of grid points with rain rates above a given threshold. 3. Skill Score for Probabilities 2. Probabilities Q: On which spatial scales does the forecast resemble the observation? FBS worst  no colocation of non-zero fractions 0 threshold

5 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff … Fraction Skill Score (FSS) (Roberts and Lean, 2005) 4. Useful Scales useful scales are marked in bold in the graphics

6 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Fuzzy Verification - Methods Upscaling - ETSFraction Skill Score - FSS Roberts & Lean (2005) Score: FSS for fractions / probabilities (0 = mismatch, 1 = perfect match) Score: ETS (Equitable Threat Score) (-1/3 = mismatch, 1 = perfect match) „Spatial averaging damps out areas with high rain rates and spreads areas of lower rain rates.“ (B.Ebert,2009) „ETS tends to increase with white space (rare events).“ (B.Ebert,2009) „The absolute value of the FSS is less useful than the scale where an acceptable level of skill is reached.“ (Mittermaier,2008) “The score is most sensitive to rare events.” (Roberts, 2008)

7 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff good bad COSMO-7 better COSMO-2 better COSMO-2COSMO-7Difference - = - = Upscaling and Fractions Skill Score Jun – Nov 2007 Fractions skill score Upscaling

8 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff COSMO-2 vs. COSMO-7 DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better abs(median) / 0.5(q95-q05) [ 10, Inf ] [ 2, 5 [ [ 5, 10 [ [ 1, 2 [ [ 0, 1 [ Values = Score of COSMO-2 Size of numbers = abs(Median) / 0.5(q95-q05)  measure for significance of differences ¦ COSMO-2 – COSMO-7 ¦ q(50%) q(95%) q(5%) 5% 95%

9 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Wilcoxon ranked probability test???

10 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Sensitivities Only 00 and 12 UTC model runs COSMO-2 & COSMO-7: leadtimes 3-6,6-9,9-12,12-15  absolute values of COSMO-2 slightly lower, but still the same pattern of differences DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

11 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Weather type verification: COSMO # cases Threshold [mm/3h] Upscaling Fractions Skill Score Window size: 3gp, 6.6 kmWindow size: 27gp, 60 km Window size: 3gp, 6.6 km Window size: 27gp, 60 km

dx NE45 ///// N ////// NW // SE11119/// S111119// SW // E91527////45 W113915/// F 27 45/// H27 45//// L /// dx NE15/////// N//////// NW // SE11139/// S111339// SW33599/// E15/////// W33359/// F99 //// H/ ////// L3599 /// COSMO-2, gridpoints* 2.2 kmCOSMO-7, gridpoints* 6.6 km Smallest spatial scale [gridpoints] where the forecast has been useful regarding to FSS „useful scales“ definition # cases % obs gridpts >= thresh (whole period) < <0.1

13 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff % NE <0.1 N <0.1 0 NW <0.1 SE <0.1 S SW E <0.1 W <0.1 F H <0.1 L ALL <0.1 Fraction of observation gridpoints >= threshold  climatology

14 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Frequency of Weather Classes, June – November

15 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Northerly Winds (NE,N) 11 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in northerly wind situations clearly worse than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

16 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Southerly Winds (SE,S) 12 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in southerly wind situations clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

17 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Northwesterly winds (NW) 23 days COSMO-2 vs.COSMO-7 COSMO-2 (wc) vs.COSMO-2 (all)  COSMO-2 in nothwesterly wind situations at large thresholds clearly better than over whole period. DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better

18 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Flat (F) 15 days COSMO-2 vs.COSMO-7COSMO-2 (wc) vs.COSMO-2 (all) DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better DIFFERENCESDIFFERENCES COSMO-2 (all) better COSMO-2 (wc) better  COSMO-2 in flat pressure situations clearly worse than over whole period.

19 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Summary / Conlusions Fuzzy verification of the D-PHASE operations period in 2007 has shown that COSMO-2 generally performed better than COSMO-7 on nearly all scales part of this superiority is caused by the higher update frequency, but using same model runs still shows the same pattern of differences the results for different weather types show large variations best results were found for southerly winds and winds from Northwest and West, Northeasterly winds as well as Flat pressure situations lead to worse perfomance of both models

20 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff UP with other scores POFD = FA/(CN+FA) Probability of false detection (Perfect score: 0) FAR = FA/(H+FA) False Alarm Ratio (Perfect score: 0) POD = H/(H+M) Probability of Detection (Perfect score: 1) BIAS = (H+FA) / (H+M) Frequency Bias (perfect score: 1) HK =POD – FAR True Skill Statistik (perfect score: 1) OR = (H*CN) / (M*FA) Odds Ratio (perfect score: infinity)

21 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff dx NE N NW SE S SW E W F H L dx NE N NW SE S SW E W F H L COSMO-2, gridpoints* 2.2 kmCOSMO-7, gridpoints* 6.6 km UP „useful scales“? How to define usefuls scales for ETS?

22 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Fuzzy Verifikation Intensity Scale (IS) (Casati et al. 2004) Transformation of Fcst and Obs into binary images on a rain/no-rain basis for the rainfall rate thresholds. Difference between forecast and observation = binary error image. Decomposition into the sum of components at different spatial scales by performing a two dimensional discrete wavelet decomposition. Binary forecastBinary observation Score: Mean squared error (MSE) and MSE skill score (SS) for each spatial scale component of the binary error image. What is the relative improvement of the forecast over some reference forecast?

23 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Intensity Scale COSMO-2COSMO-7

24 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Intensity Scale COSMO-2

25 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Outlook operational Fuzzy verification is about to start, including Upscaling and Fraction Skill Score season, year weather-type dependant Intensity scale results will further be investigated (new developments of B. Casati)

26 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff All Weatherclasses Overview over all weatherclasses – differences COSMO-2 minus COSMO-7 and absolute values of COSMO-2, here without bootstrapping

27 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Northerly Winds (NE,N,NW) NNW NE 23 days 4 days7 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

28 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Southerly Winds (SE,S,SW) S SE SW 7 days 45 days 5 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

29 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff East and West (E,W) EW 31 days5 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

30 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Flat, High, Low (F,H,L) FHL 34 days7 days15 days DIFFERENCESDIFFERENCES COSMO-7 better COSMO-2 better

31 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff Summary JJASONDOP Useful scales of COSMO-2 are the same for all three time periods considered: <= 0.2 mm/3h  scales larger or equal 2.2 km (= 1* gridscale ) 0.5 mm/3h  scales larger or equal 6.6 km (= 3* gridscale ) 1.0 mm/3h  scales larger or equal 19.8 km (= 9* gridscale ) 2.0 mm/3h  scales larger or equal 33.0 km (= 15* gridscale ) 5.0 mm/3h  scales larger or equal 99.0 km (= 45* gridscale ) Most prominent advantage for COSMO- 2! Differences are significant. COSMO-2 better but on many scales only slightly. Advantages on small scales/all thresholds and large thresholds/large scales. Differences are significant. COSMO-2 better than COSMO-7 on all scales. Differences are significant. Most frequent weatherclasses: SW (32 days) and W (21 days). Most frequent weatherclasses: H (24 days) and NW (17 days). Most frequent weatherclasses: SW (45 days) and H (34 days).

32 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff JJASONDOP NE -Relatively low values for UP -FSS: COSMO-2 better for thresholds <= 2 mm/3h, useful scales only for largest window size (99 km) - COSMO-2 better on nearly all scales, especially for thresholds of 2-5 mm/3h. Large threshold gave no result in UP  only small - scale events (averaging damps values out). Apparently, COSMO-2 could simulate those events quite well. -Low absolute values (FSS < 0.6, UP < 0.3), no useful scales -COSMO-2 clearly better for thresholds <= 2 mm/3h (FSS) - UP: the averaged precipitation was equally well represented in both models. Only 2 mm/3h threshold better captured in COSMO-2. N Only few days – no clear superiority of one model. Absolute values relatively low. No useful scales. NW - COSMO-2 performed better than COSMO-7, especiall for large thresholds (> 1 mm/3h, FSS and UP). -Many days with that weather type -Over most scales rarely different behaviour -Thresholds 5-10 mm/3h better in COSMO- 2 as well as spatial scale of 6.6 km - COSMO-2 performed better than COSMO-7, especiall for large thresholds (>= 5 mm/3h, FSS and UP). Summary Weatherclasses I

33 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff JJASONDOP SE -Only few days; similar behaviour in all time periods: COSMO-2 better, especially for thresholds around 1 mm/3h and small spatial scales (FSS) and large spatial scales (UP) -Large absolute values (e.g. FSS up to 0.9, UP up to 0.66 for DOP) S - Only few days; COSMO-2 better on most scales, most clearly for low thresholds concerning averaging (UP) and large thresholds and small spatial scales regarding probabilities (FSS), in JJA COSMO-2 clearly better also for smallest eindow size (6.6 km) for all thresholds SW - Clearly better performance of COSMO-2, especially on smaller spatial scales - Similar performance of both models, but COSMO-2 better for moderate thresholds (1-2 mm/3h) - Clearly better performance of COSMO-2, especially on smaller spatial scales Summary Weatherclasses II

34 Weather type dependant fuzzy verification | COSMO GM, Tanja Weusthoff JJASONDOP E -Only few cases (5 for DOP), COSMO-2 clearly better for low thresholds (<= 1 mm), while for a threshold of 2 mm/3h COSMO-7 better on largest spatial scales -Rather low absolute values W - COSMO-2 clearly better on all scales, especially for thresholds <= 2 mm and smaller spatial scales - COSMO-2 slightly better only on small spatial scales - COSMO-2 better for thresholds <= 2 mm and especially smaller spatial scales F -Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds. -Comparable low absolute values, rarely difference in the performance of the two models. -COSMO-2 slightly better for small thresholds - Comparable low absolute values, but COSMO-2 clearly better than COSMO-7, especially for medium and large precipitation thresholds. H -very low absolute values (especially in SON), but COSMO-2 clearly better than COSMO-7 for precipitation thresholds <= 2 mm/3h L - Only few cases (7 for DOP), COSMO-2 better for small spatial scales and thresholds <= 2 mm/3h Summary Weatherclasses III