Verification methods - towards a user oriented verification The verification group

COSMO GM Cracow September 2008 Course of verification on principal Verification Observation Forecast Data control Enduser Modeller Analysis Data control

COSMO GM Cracow September 2008 What are the results of verification? RMSE | S1 |ANOC ETS FBI FSS BIAS BSS ROC ISS

COSMO GM Cracow September 2008 What is a good forecast? A.H. Murphy, Weather and Forecasting, 8(1993), Iss.2,  Type 1 of goodness: Consistency  Correspondence between forecasts and judgements  Type 2 of goodness: Quality  Comparison of results for different seasons  Type 3 of goodness: Value  Incremental benefits of forecasts to users

COSMO GM Cracow September 2008 What is a good forecast? A.H. Murphy, Weather and Forecasting, 8(1993), Iss.2,  Type 1 of goodness: Consistency  Correspondence between forecasts and judgements  Type 2 of goodness: Quality  Comparison of results for different reasons  Type 3 of goodness: Value  Incremental benefits of forecasts to users

COSMO GM Cracow September 2008 Attributes of a forecasts related to observations (I)  BIAS  Confidence between mean forecast and mean observation (BIAS, FBI)  Association  Overall strength of linear relationships between individual pairs of observation and forecast (correlation)  Accuracy  Average correspondence between individual pairs of observation and forecast (mean absolute error, root mean square error, Brier score,...)  Skill  Accuracy of a forecast relative to a reference forecast (reduction of variance, Brier skill score)  Reliability  Correspondence between conditional mean observation and and conditioning forecast averaged over all forecasts (elements of conditional verification, probability forecasts)

COSMO GM Cracow September 2008 Attributes of a forecasts related to observations (II)  Resolution  Difference between conditional mean obseration and unconditional mean observation averaged over all forecasts (parts of the Brier score and the mean square error, probability forecasts)  Sharpness  Variability of forecasts as described by distribution of forecasts (parameter of forecast distribution: mean, standard deviation, skewness, kurtosis, median, quantiles,..., probability forecasts)  Discrimination I  Correspondence between conditional mean forecast and conditioning observation averaged over all observations (elements of conditional verification, probability forecasts)  Discrimination II  Difference between conditional mean forecast and unconditional mean forecast averaged over all obsercations (elements of conditional verification probability forecasts, )  Uncertainty  Variability of observations as described by distribution of observations (parameter of observation distribution: mean, standard deviation, skewness, kurtosis, median, quantiles,...)

COSMO GM Cracow September 2008 Attributes of a forecasts related to observations(I)  Bias - the correspondence between the mean forecast and mean observation.  Association - the strength of the linear relationship between the forecasts and observations (for example, the correlation coefficient measures this linear relationship)  Accuracy - the level of agreement between the forecast and the truth (as represented by observations). The difference between the forecast and the observation is the error. The lower the errors, the greater the accuracy.  Skill - the relative accuracy of the forecast over some reference forecast. The reference forecast is generally an unskilled forecast such as random chance, persistence (defined as the most recent set of observations, "persistence" implies no change in condition), or climatology. Skill refers to the increase in accuracy due purely to the "smarts" of the forecast system. Weather forecasts may be more accurate simply because the weather is easier to forecast -- skill takes this into account. referring to: A.H. Murphy, Weather and Forecasting, 8(1993), Iss.2,

COSMO GM Cracow September 2008  Reliability - the average agreement between the forecast values and the observed values. If all forecasts are considered together, then the overall reliability is the same as the bias. If the forecasts are stratified into different ranges or categories, then the reliability is the same as the conditional bias,  Resolution - the ability of the forecast to sort or resolve the set of events into subsets with different frequency distributions. This means that the distribution of outcomes when "A" was forecast is different from the distribution of outcomes when "B" is forecast. Even if the forecasts are wrong, the forecast system has resolution if it can successfully separate one type of outcome from another. Attributes of a forecasts related to observations(II)

COSMO GM Cracow September 2008  Sharpness - the tendency of the forecast to predict extreme values. To use a counter-example, a forecast of "climatology" has no sharpness. Sharpness is a property of the forecast only, and like resolution, a forecast can have this attribute even if it's wrong (in this case it would have poor reliability).  Discrimination - ability of the forecast to discriminate among observations, that is, to have a higher prediction frequency for an outcome whenever that outcome occurs.  Uncertainty - the variability of the observations. The greater the uncertainty, the more difficult the forecast will tend to be. Attributes of a forecasts related to observations(III)

COSMO GM Cracow September 2008 Current focal points of verification  Spatial verification methods  object oriented methods  „fuzzy“- techniques  Verification of probabilistic and ensemble forecasts  ensemble pdf  generic probability forecasts  probability of an event  Verification of extreme (rare) events  high-impact events  Operational verification  evaluation and monitoring  User-oriented verification strategies  tailored verification for any user  Forecast value  cost - loss analysis, development of an universal score  Verification packages  VERSUS, MET

COSMO GM Cracow September 2008 Some problems concerning significance  Most scores have a statistical outfit and seem to be open for significance tests.  Traditional significance tests require a defined number of degrees of freedom.  In most cases observations, forecasts and errors are correlated.  Therefore, the degrees of freedom cannot be obtained easily.  One way out: resampling and bootstrapping  What about statistical significance and meteorological significance?

COSMO GM Cracow September 2008 User-oriented verification strategies  What is the interest of an user?  Administrator:  Did forecasts yield to better results during last period of interest and in general?  What type of focal points for model development are of current interest? ...  Modeller  What type of errors occur in general?  What are the reasons for such errors?  How should the model modified in order to avoid or to reduce these errors?  If I have found the reason(s) for the error(s) and I have reduced their effect(s), is the forecast then improved? ...  External user  How can I interpret the forecasts?  What is the benefit of forecasts for me? ...

COSMO GM Cracow September 2008 User-oriented verification step by step 1. Diagnosis of errors normally done by examining the BIAS or the FBI The problem - mean values of observed and forecasted T2m over Germany during Sommer 2005 and Winter 2005/2006 (RMSE/STDV) Summer 2005 Winter 2005/2006

COSMO GM Cracow September 2008 User-oriented verification step by step 1. Diagnosis of errors normally done by examining the BIAS or the FBI The problem - mean values of observed and forecasted gusts over Germany during Spring 2007 (RMSE/STDV) COSMO-DE COSMO-EU

COSMO GM Cracow September 2008 User-oriented verification step by step 1. Diagnosis of errors normally done by examining the BIAS or the FBI Examples for four scores in four stylisized situations:

COSMO GM Cracow September 2008 An example for conditional verification Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 (RMSE and STDV) Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 observed and forecasted values lower than 1020 hPa (RMSE and STDV) Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 observed and forecasted values higher than 1020 hPa (RMSE and STDV)

COSMO GM Cracow September 2008 User-oriented verification step by step 2. Some changes made by modellers  New diagnosis of gusts  to reduce the overestimation of gusts:  use wind at 10 m instead of interpolated wind from 30 m to compute gusts  New diagnosis of temperature 2m  to reduce the strong negative bias during winter and get a more realistic diurnal cycle:  set z 0 to 2 cm over land and change some other parameterizations  New SSO scheme (currently under examination)

COSMO GM Cracow September 2008 User-oriented verification step by step 3. The effects  New diagnosis of gusts  The overestimation of gusts is now reduced.  But: Extreme gusts are underestimated.  New diagnosis of temperature 2m  to reduce the strong negative bias during winter and get a more realistic diurnal cycle:  set z 0 to 2 cm over land and change some other parameterizations  New SSO scheme (currently under examination)

COSMO GM Cracow September 2008 User-oriented verification step by step 4. The proof of the effects: New diagnosis of gusts gusts > 12 ms -1 Böenverifikation der Experimente 6278 (COSMO-EU) Operational run 6301 (COSMO-DE) ETS FBI

COSMO GM Cracow September 2008 User-oriented verification step by step 4. The proof of the effects: New diagnosis of temperature 2m Comparison of COSMO-EU with experiment UTC: April/June 2007 : RMSE COSMO-EU area

COSMO GM Cracow September 2008 A basic law during model development: There are no gains without any losses! (maybe with some exceptions) Therefore, one has to look both at benefits and risks.

COSMO GM Cracow September 2008 One of known exceptions: The effect of a SSO scheme in COSMO-EU

COSMO GM Cracow September 2008 One of known exceptions: The effect of a SSO scheme in COSMO-EU

COSMO GM Cracow September 2008 User-oriented verification step by step 5. The risk: New diagnosis of gusts gusts > 25 ms -1 ETS FBI Böenverifikation der Experimente 6278 (COSMO-EU) Operational run 6301 (COSMO-DE)

COSMO GM Cracow September 2008 User-oriented verification step by step 5. The risk: New diagnosis of temperature 2m Comparison of COSMO-EU with experiment UTC: April/June 2007 : BIAS COSMO-EU area

COSMO GM Cracow September 2008 User-oriented verification step by step 6. The operational effect: New diagnosis of temperature 2m

COSMO GM Cracow September 2008 User-oriented verification step by step 7. The effect for administrators experessed in „The Score COSI“

COSMO GM Cracow September 2008 User-oriented verification step by step 7. The effect for administrators experessed in „The Score COSI“ asr Prognostic cloud ice Prognostic precipitation LME V 3.19 V 3.22 T 2m

COSMO GM Cracow September 2008 User-oriented verification step by step 8. Questions Are there any questions to me? Question from the verification group: What are the requirements to the verification process by users in order to make the process of model development as effective as possible?

COSMO GM Cracow September 2008 User-oriented verification step by step 7. The effect for administrators experessed in „The Score COSI“ asr

COSMO GM Cracow September 2008 User-oriented verification step by step 7. The effect for administrators experessed in „The Score COSI“