1. Experiences concerning fuzzy-verification and pattern recognition methods Ulrich Damrath

2. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 2 Outlook Results on operational verifcation for winter and summer month An approach concerning significance test of fuzzy-verification results Estimation of consistency of forecasts using a pattern recognition method (CRA method by Beth Ebert)

3. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 3 Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours GMECOSMO-EU COSMO-DE

4. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 4 ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours GMECOSMO-EU COSMO-DE

5. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 5 Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours GlobalEurope Germany

6. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 6 ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours GlobalEurope Germany

7. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 7 Examination of statistical significance of fuzzy-verification results using bootstrapping Basic idea of bootstrapping: Repeat a resampling all elements of a given in a sample of forecasts and observations as often as necessary (N times) and calculate the relevant score(s) Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles Application to fuzzy-verification Resampling is done using blocks. Blocks are defined as single days. Number of resampling cases: N=Days*100 Calculation scores from N samples for NT thesholds and NW windows Calculation of quantiles for each window and threshold

8. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 8 Values and quantiles 0.1 and 0.9 for Upscaling ETS GME, period June - August 2009 Germany

9. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 9 Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-EU, period June - August 2009 Germany

10. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 10 Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009 Germany

11. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 11 Next step: Evaluation of significance First impression: Is the result of Model 1 better than the result of Model 2? Significance hypothesis checked using a Wilcoxon-test (IDL-code RS_TEST)

12. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 12 Differences between GME and COSMO-EU ETS(COSMO-EU) - ETS(GME)Significance test COSMO-EU better than GME COSMO-EU worse than GME Germany

13. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 13 Differences between GME and COSMO-DE ETS(COSMO-DE) - ETS(GME)Significance test COSMO-DE better than GME COSMO-DE worse than GME Germany

14. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 14 Differences between COSMO-DE and COSMO-EU ETS(COSMO-DE) - ETS(COSMO-EU) Significance test COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU Germany

15. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 15 ETS(COSMO-DE) - ETS(COSMO-EU) COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU Significance test Differences between COSMO-DE and COSMO-EU

16. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 16 Example of good precipitation forecast of COSMO-DE Zeigten die numerischen Modelle und die statistischen Prognose- --------------------------------------------------------------- verfahren Signale für das Ereignis? ----------------------------------- Die Numerik zeigte im Vorfeld vermehrt Signale für kräftige Konvektion. Während diese bei GME und COSMO-EU recht breit gestreut und pauschal auftraten, signalisierten mehrere COSMO- DE-Läufe eine linienartige Struktur mit unwetterartigen Zellen (auf Basis der 1- bzw. 3-stündigen RR-Prognosen) im Grenzbereich von Hessen zu NRW und Niedersachsen. Diese Linie trat dann in den Mittags- und frühen Nachmittagsstunden tatsächlich auf, wenn auch nicht 100%ig kongruent, aber doch in der Nähe, so dass in diesem Fall von einer guten Prognose gesprochen werden kann (mehr dazu siehe "Zentraler UW- Sofortbericht" der VBZ).

17. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 17 Example of good precipitation forecast of COSMO-DE 3h-precipitation forecast of COSMO-DE valid 10.08.2009 12 UTC, left 03 UTC +9h, right 06 UTC +6h. 3h-precipitation observation 10.08.2009 09-12 UTC

18. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 18 Example of good precipitation forecast of COSMO-DE compared to other models

19. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 19 Example of good precipitation forecast of COSMO-DE compared to other models

20. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 20 About consistency and inconsistency Forecasters are interested in consistent model forecasts. But due to growing of errors during forecast time forecasts consistency cannot be expected concerning all properties of the forecasted fields! Inconsistency: Differences between forecasts that are valid for the same time concerning different properties of the forecasted fields (properties of the pattern, values at special points of interest, extreme values,...) Differences between the forecasted fields concerning phase, amplitude and the remaining part

21. Entity-based QPF verification (rain blobs) by E. Ebert (BOM Melbourne) Verify the properties of the forecast rain system against the properties of the observed rain system: location rain area rain intensity (mean, maximum) ObservedForecast CRA error decomposition The total mean squared error (MSE) can be written as: MSE total = MSE displacement + MSE volume + MSE pattern Configuration for the current study: - Observations: forecasts: 06-30 hours - Forecasts : forecasts: 30-54 hours and forecasts: 54-78 hours

22. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 22 Dark :forecasts 30-54 h Light:forecasts 54-78 h

23. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 23 Dark :forecasts 30-54 h Light:forecasts 54-78 h

24. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 24 Dark :forecasts 30-54 h Light:forecasts 54-78 h

25. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 25 Dark :forecasts 30-54 h Light:forecasts 54-78 h

26. U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach 2009 26 Summary Scores like Fractions skill score and ETS from upscaling show in general advantages of COSMO models compared to GME. This is true especially for summer months. For winter months all models have nearly the same quality for low precipitation amounts and large window sizes for averaging. Significance test lead to the results, that: The advantages of COSMO models compared to GME are statistically significant for most window sizes and precipitation amounts. The differences between COSMO-EU and COSMO-DE are not significant altough there are systematical differences for different precipitation amounts and window sizes. There are some cases with very useful precipitation forecasts of COSMO-DE compared to COSMO-EU from the view of forecasters. A study about the consistency of precipitation forecasts showed - it could be expected, but now it is proved - that: Forecasts of high precipitation amounts are less consistent than those for low precipitation amounts. Pattern errors contribute most to forecast errors. During winter months volume errors are higher than displacement errors. During summer months displacement errors are higher than volume errors.