Experiences concerning fuzzy-verification and pattern recognition methods Ulrich Damrath

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Outlook Results on operational verifcation for winter and summer month An approach concerning significance test of fuzzy-verification results Estimation of consistency of forecasts using a pattern recognition method (CRA method by Beth Ebert)

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time hours GMECOSMO-EU COSMO-DE

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time hours GMECOSMO-EU COSMO-DE

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time hours GlobalEurope Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time hours GlobalEurope Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Examination of statistical significance of fuzzy-verification results using bootstrapping Basic idea of bootstrapping: Repeat a resampling all elements of a given in a sample of forecasts and observations as often as necessary (N times) and calculate the relevant score(s) Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles Application to fuzzy-verification Resampling is done using blocks. Blocks are defined as single days. Number of resampling cases: N=Days*100 Calculation scores from N samples for NT thesholds and NW windows Calculation of quantiles for each window and threshold

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Values and quantiles 0.1 and 0.9 for Upscaling ETS GME, period June - August 2009 Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-EU, period June - August 2009 Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009 Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Next step: Evaluation of significance First impression: Is the result of Model 1 better than the result of Model 2? Significance hypothesis checked using a Wilcoxon-test (IDL-code RS_TEST)

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Differences between GME and COSMO-EU ETS(COSMO-EU) - ETS(GME)Significance test COSMO-EU better than GME COSMO-EU worse than GME Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Differences between GME and COSMO-DE ETS(COSMO-DE) - ETS(GME)Significance test COSMO-DE better than GME COSMO-DE worse than GME Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Differences between COSMO-DE and COSMO-EU ETS(COSMO-DE) - ETS(COSMO-EU) Significance test COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU Germany

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach ETS(COSMO-DE) - ETS(COSMO-EU) COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU Significance test Differences between COSMO-DE and COSMO-EU

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Example of good precipitation forecast of COSMO-DE Zeigten die numerischen Modelle und die statistischen Prognose verfahren Signale für das Ereignis? Die Numerik zeigte im Vorfeld vermehrt Signale für kräftige Konvektion. Während diese bei GME und COSMO-EU recht breit gestreut und pauschal auftraten, signalisierten mehrere COSMO- DE-Läufe eine linienartige Struktur mit unwetterartigen Zellen (auf Basis der 1- bzw. 3-stündigen RR-Prognosen) im Grenzbereich von Hessen zu NRW und Niedersachsen. Diese Linie trat dann in den Mittags- und frühen Nachmittagsstunden tatsächlich auf, wenn auch nicht 100%ig kongruent, aber doch in der Nähe, so dass in diesem Fall von einer guten Prognose gesprochen werden kann (mehr dazu siehe "Zentraler UW- Sofortbericht" der VBZ).

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Example of good precipitation forecast of COSMO-DE 3h-precipitation forecast of COSMO-DE valid UTC, left 03 UTC +9h, right 06 UTC +6h. 3h-precipitation observation UTC

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Example of good precipitation forecast of COSMO-DE compared to other models

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Example of good precipitation forecast of COSMO-DE compared to other models

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach About consistency and inconsistency Forecasters are interested in consistent model forecasts. But due to growing of errors during forecast time forecasts consistency cannot be expected concerning all properties of the forecasted fields! Inconsistency: Differences between forecasts that are valid for the same time concerning different properties of the forecasted fields (properties of the pattern, values at special points of interest, extreme values,...) Differences between the forecasted fields concerning phase, amplitude and the remaining part

Entity-based QPF verification (rain blobs) by E. Ebert (BOM Melbourne) Verify the properties of the forecast rain system against the properties of the observed rain system: location rain area rain intensity (mean, maximum) ObservedForecast CRA error decomposition The total mean squared error (MSE) can be written as: MSE total = MSE displacement + MSE volume + MSE pattern Configuration for the current study: - Observations: forecasts: hours - Forecasts : forecasts: hours and forecasts: hours

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Dark :forecasts h Light:forecasts h

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Dark :forecasts h Light:forecasts h

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Dark :forecasts h Light:forecasts h

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Dark :forecasts h Light:forecasts h

U. Damrath: Experiences concerning fuzzy-verification... - COSMO GM, Offenbach Summary Scores like Fractions skill score and ETS from upscaling show in general advantages of COSMO models compared to GME. This is true especially for summer months. For winter months all models have nearly the same quality for low precipitation amounts and large window sizes for averaging. Significance test lead to the results, that: The advantages of COSMO models compared to GME are statistically significant for most window sizes and precipitation amounts. The differences between COSMO-EU and COSMO-DE are not significant altough there are systematical differences for different precipitation amounts and window sizes. There are some cases with very useful precipitation forecasts of COSMO-DE compared to COSMO-EU from the view of forecasters. A study about the consistency of precipitation forecasts showed - it could be expected, but now it is proved - that: Forecasts of high precipitation amounts are less consistent than those for low precipitation amounts. Pattern errors contribute most to forecast errors. During winter months volume errors are higher than displacement errors. During summer months displacement errors are higher than volume errors.