1. COSMO General Meeting 2011 WG5 Parallel Session 5 September 2011
Verification results at MeteoSwiss in the year Francis Schubiger, Vanessa Stauch, Tanja Weusthoff, Pirmin Kaufmann MeteoSwiss
COSMO General Meeting 2011
WG5 Parallel Session 5 September 2011

2. Content results of the operational verification in 2011
separate verification of all eight daily COSMO-2 forecasts
neighborhood verification (also weather type dependant)
COSMO-2 03 UTC forecasts up to +48h
extension of Beth Ebert‘s package: -> „fuzzy in time“

3. Verification of COSMO-2, COSMO-7, COSMO-LEPS and IFS
Operational (COSMO-2, COSMO-7, COSMO-LEPS [mean], IFS):
Surface (3h-steps SYNOP and 1h-steps SMN [Swiss Met Net])
psred, ps, T2m, Td2m, 10m-wind, cloud cover, gusts, precipitation (1h, 12h), global radiation
Upper-air (TEMP)
T, RH, wind, F
Quasi-operational (COSMO-2, COSMO-7, IFS):
precipitation with RADAR (neighborhood/fuzzy verification, Ebert’s package) also weather-type dependant
Monitoring: SMN (swiss SYNOPs) , RADAR, windprofilers
Ongoing in experimental modus:
windprofiler over Switzerland
fluxes at Payerne
New / Next developments:
VAD (locations of swiss radars)
=> All these packages should be replaced by VERSUS 2
in red: new developments in 2010

4. T2m: mean diurnal cycle (first 24h forecasts) domain Switzerland (hourly SYNOP‘s)
Autumn 2010
Spring 2011
OBS
COSMO-7
COSMO-2
Winter 2010/2011
Summer 2011
V. Stauch, P. Kaufmann

5. T2m: mean diurnal cycle Spring 2011 COSMO-7 vs IFS SYNOP‘s over COSMO-7 domain
OBS
COSMO-7
IFS
V. Stauch, P. Kaufmann

6. T2m: mean diurnal cycle Spring 2011 COSMO-7 vs COSMO-LEPS SYNOP‘s over COSMO-7 domain
OBS
COSMO-7
COSMO-LEPS
V. Stauch, P. Kaufmann

7. Global radiation (inclined): Spring 2011 COSMO-7 mean diurnal cycle SYNOP‘s over Switzerland
The mean temperature of the COSMO-LEPS mean is generally too low resulting in negative biases of between 0 and -1.5K for most European stations, and ~ -2K for Alpine stations. This results from the underestimated daily cycle with a too low temperature maximum. Therefore, the standard deviation of the error for COSMO-LEPS mean is higher when compared to COSMO-7, in particular during the daytime.
V. Stauch, P. Kaufmann

8. Global radiation (horizontal): Spring 2011 COSMO-7 mean diurnal cycle SYNOP‘s over Switzerland
The mean temperature of the COSMO-LEPS mean is generally too low resulting in negative biases of between 0 and -1.5K for most European stations, and ~ -2K for Alpine stations. This results from the underestimated daily cycle with a too low temperature maximum. Therefore, the standard deviation of the error for COSMO-LEPS mean is higher when compared to COSMO-7, in particular during the daytime.
V. Stauch, P. Kaufmann

9. Precipitation: Spring 2011 geographical distribution
precipitation 12h sums
frequency bias (1 mm/12h)
Spring 2011
COSMO & +48h
V. Stauch, P. Kaufmann

10. Precipitation (12h-sums +12 to +24h): Spring 2011 over Switzerland (SYNOP‘s) frequency bias: COSMO-7 & COSMO-2
observed frequency
Comparison of the frequency bias (FBI) for COSMO-2 and COSMO-7 over Switzerland as a function of thresholds for the lead time range 13-24h. The observed frequency (OF) is plotted underneath. (lines between the data points are added for better overview and are not meant to be meaningful interpolations)
V. Stauch

11. Precipitation (12h-sums +36 to +48h): Spring 2011 over Switzerland (SYNOP‘s) frequency bias: COSMO-7 & IFS
observed frequency
Comparison of the frequency bias (FBI) for COSMO-2 and COSMO-7 over Switzerland as a function of thresholds for the lead time range 13-24h. The observed frequency (OF) is plotted underneath. (lines between the data points are added for better overview and are not meant to be meaningful interpolations)
V. Stauch

12. Precipitation (12h-sums +12 to +24h): Spring 2011 over Switzerland (SYNOP‘s) COSMO-7 & COSMO-2
for both models
mean over 9 gridpoints for
each station
V. Stauch

13. Precipitation (12h-sums): Spring 2011 over Switzerland (SYNOP‘s) COSMO-7 & IFS
V. Stauch

14. Hourly precipitation: diurnal cycle
Swiss SYNOPs (1h acc)
CH-radarcomposit (3h acc)
SMN
COSMO-7
COSMO-7
July 2010
COSMO-2
COSMO-2
RADAR
fcst: h
fcst: h
August 2010
RADAR
COSMO-7
COSMO-2
SMN
fcst: h
fcst: h
V. Stauch

15. Verification of surface weather parameters
2m-temperature:
cold bias in winter and warm bias in summer
positive bias more pronounced during nighttime
COSMO-2 ~0.5 K warmer (higher values) than COSMO-7
2m-dewpoint:
negative bias (~1 K) in the summer and autumn period
10m-windspeed:
negative bias along the coast and on mountains, positive bias inland
total cloudiness:
mean daily cycle not well represented (mainly overestimation during night and underestimation during day)
precipitation:
overestimation (except in Summer ~ no bias) most pronounced in Winter (over COSMO-2 domain: 30-35% ; over Swizerland ~45%)
higher amounts in COSMO-2 / higher amounts over Alpine area
low amounts (0.1 mm/12h): overestimated in Winter (over the Alps all seasons)
high amounts (10 mm/12h): underestimated in COSMO-7 (over full domain by ~30% in summer, 20% in winter)

16. TEMPS verification: Windspeed +24h Winter 10/11
all TEMPs
COSMO-2
domain
COSMO-7
COSMO-2

17. Verification with vertical profiles: main results
temperature: cold bias (~ 0.5 K) in winter and warm bias (~ 0.5 K) in summer from ground up to 300 hPa
windspeed: positive bias in PBL (up to +1 m/s in winter) and slight negative bias above 400 hPa

18. Separate verification of the eight daily COSMO-2 forecasts
questions of interest:
which is the best forecast for the afternoon/evening convection ? – is it the latest forecast or perhaps a forecast with a longer „spin up“ time ?
are there significant differences between the eight forecasts ?
period: COSMO-2 forecasts from 1 May to 31 July 2011
Verification with hourly obs over Switzerland

19. COSMO-2: precipitation diurnal cycle forecasts of 00, 03, 06 and 09 UTC
Vanessa Stauch

20. COSMO-2: precipitation diurnal cycle forecasts of 12, 15, 18 and 21 UTC
Vanessa Stauch

21. COSMO-2: precipitation diurnal cycle all eight daily forecasts
Vanessa Stauch

22. COSMO-2: 2m-temperature diurnal cycle all eight daily forecasts
Vanessa Stauch

23. COSMO-2: 2m-temperature bias all eight daily forecasts
Vanessa Stauch

24. COSMO-2: 10m-wind speed diurnal cycle all eight daily forecasts
Vanessa Stauch

25. Neighborhood (fuzzy) verification for precipitation
verification on lower resolution: avoid the „double penalty problem“
method
Original data
“fuzzyfication”
score
example
Upscaling
Average
Equitable threat score
Fractions Skill Score (Roberts and Lean, 2005)
Fractional coverage
Skill score with reference to worst forecast X x X x
T. Weusthoff

26. How to read „fuzzy grafics“
best skill for low precipitation on large scales
numbers / colours
 value of the score, here: FSS
bad
good
increasing window size
numbers in bold
 useful scales (only for FSS)
low skill for high precipitation on small scales
Increasing threshold
T. Weusthoff, MeteoSwiss

27. Neighborhood verification for precipitation
results for 2010
3h accumulated precipitation sums
over the domain of the swiss radar composit
models: COSMO-2 and COSMO-7
for all 8 daily forecast runs, precipitation sums from +3 to +6h
observation
precipitation estimates of the swiss radar composit
in case of a missing value, the full date will not be evaluated

28. - = - = Fractions Skill Score Upscaling
Neighbourhood Verification, January–December h sums ( h) Fractions Skill Score (top) and Upscaling (bottom)
Fractions Skill Score - =
COSMO-2
COSMO-7
COSMO-2 - COSMO-7
Upscaling - =
Zunehmender Skill mit räumlicher Skala, abnehmender Skill mit zunehmendem Threshold…
COSMO-2 besser als COSMO-7 auf nahezu allen Skalen
T. Weusthoff
good
bad
COSMO-7 better
COSMO-2 better

29. FSS, COSMO-2 minus COSMO-7, 2010
Winter
Spring
COSMO-2 better
COSMO-7 better
numbers = FSS-Score of COSMO-2
colours = differences COSMO-2 minus COSMO-7
COSMO-2: similar skill in all seasons
COSMO-2 better than COSMO-7 on almost all scales
COSMO-2 better than COSMO-7 especially in Winter and Summer
Summer
Autumn
Tanja Weusthoff

30. Achtung: FSS, COSMO-2 minus COSMO-7, 2009 Winter (DJF) Frühling (MAM)
COSMO-2 better
COSMO-7 better
Zahlen = FSS-Score von COSMO-2
Farben = Differenzen COSMO-2 minus COSMO-7
COSMO-2: vergleichbaren Skill in allen Saisons für kleine Schwellenwerte
hohe Schwellenwerte (ge 5mm/3h) besserer Skill in Frühling / Sommer
Frühling und Sommer: COSMO-2 grösster Vorteil im Vergleich zu COSMO-7
Herbst und Winter: COSMO-2 leicht besser auf nahezu allen Skalen
Sommer (JJA)
Herbst (SON)
Tanja Weusthoff

31. Dependency of the lead time
+3 bis +6h
+12 bis +15h
+21 bis +24h
Fractions Skill Score
Upscaling - ETS
Tanja Weusthoff
COSMO-7 better
COSMO-2 better

32. YEAR 2010 NE
(11x) S
(10x)
COSMO-7 better
COSMO-2 better
differences in Fractions Skill Score for weather-type dependant verif
COSMO-2 minus COSMO-7 F
(78x) N
(18x) SW
(49x) H
(73x) E
(4x) NW
(38x) W
(56x) SE
(4x) L
(25x)

33. Summary neighbourhood Verifikation precipitation in 2010
COSMO-2 better than COSMO-7 on all scales, differences become less with increasing leadtime
good forecast of the spatial structure on higher scales
The skill of the models varies for different weather types and the differences between COSMO-2 and COSMO-7 varies also: - best skill: Autumn and Spring, south to northwest weather types - greatest difference COSMO-2 minus COSMO-7: Summer and Winter, north- and east types, convective cases
Tanja Weusthoff

34. COSMO-2 03 UTC up to +48h: Comparison COSMO-2 to COSMO-7
period (full year): 1 June 2010 – 31 May 2011
Main interest:
advance of COSMO-2 as compared to COSMO-7
question: “are the COSMO-2 forecasts at +48h still better than those of COSMO-7” ?
Verification with:
SYNOP over Switzerland (hourly): comparison +0 to +24h vs +24 to +48h
Swiss RadarComposit: neighborhood Verifikation for four 12h-Summen (0..+12, , , h)
TEMPs for +21h and +45h

35. Verification SYNOP-CH: 2m-temperature COSMO-2 vs COSMO-7
BIAS
VAL
advance of COSMO-2 also in the second day
MAE
STDE
V. Stauch

36. Verification SYNOP-CH: precipitation COSMO-2 vs COSMO-7 [+0 to +24h]
0.1 mm/h
1.0 mm/h
10 mm/h
Frequency bias (mm/h)
COSMO-2
COSMO-7
V. Stauch

37. Verification SYNOP-CH: precipitation COSMO-2 vs COSMO-7 [+24 to +48h]
0.1 mm/h
1.0 mm/h
10 mm/h
Frequency bias (mm/h)
COSMO-2
COSMO-7
V. Stauch

38. Verification with swiss radarcomposit
daytime: UTC
nighttime: UTC h
first forecast day h
COSMO-2
COSMO-7
COSMO-2
COSMO-7 h
second forecast day h
Overerstimation of COSMO-2
COSMO-2
COSMO-7
COSMO-2
COSMO-7
T. Weusthoff

39. FRACTIONS SKILL SCORE: COSMO-2 – COSMO-7 precipitation sums of 12h
COSMO-2 better
COSMO-7 better h h
T. Weusthoff

40. UPSCALING: COSMO-2 – COSMO-7 precipitation sums of 12h; score: ETS (equitable threat score)
COSMO-2 better
COSMO-7 better h h
T. Weusthoff

41. UPSCALING: COSMO-2 – COSMO-7 precipitation sums of 12h; score: frequency bias (absolute)
T. Weusthoff

42. UPSCALING: COSMO-2 – COSMO-7 precipitation sums of 12h; score: frequency bias (relative)
T. Weusthoff

43. 3. „Fuzzy in Time“
Extension of the spatial window with a window in time  volume (dx * dy * dt)
Evaluation of the forecasts in this volume
Time-window ntm = [1h,3h,5h,7h,9h] dx dy dt
T. Weusthoff

44. model dx dy observation dx dy T. Weusthoff t0+1 t0 e.g. FSS
fraction blue pxiels model = 12/75 (dt=3), 5/25 (dt=1) fraction blue pxiels obs = 12/75 (dt=3), 3/25 (dt=1)
t0-1
observation dx dy
t0+1 t0
T. Weusthoff
t0-1

45. hourly accumulated precipitation20 21 22 23 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 …
00-02
01-03
3 hourly accumulated precipitation
02-04
20-22
03-05
21-23
04-06
22-00
05-07
23-01
Evaluation of 3h accumulations, time window ntm = 1
09 UTC
00 UTC
03 UTC
06 UTC
23 UTC
01 UTC
02 UTC
04 UTC
Evaluation of 3h accumulations, time window ntm = 5 (t0 +-2h)
00 UTC
03 UTC
06 UTC
08 UTC
22 UTC
02 UTC
01 UTC
05 UTC
T. Weusthoff

46. FSS for different time-windows COSMO-2, July 2010
T. Weusthoff

47. FSS for different time-windows COSMO-7, July 2010
T. Weusthoff

48. FSS for different time-windows COSMO-2 minus COSMO-7, July 2010
T. Weusthoff

49. Summary „fuzzy in time“
FSS increases on all scales with increasing time-window
greatest effect for small spatical scales
lowest effect for high threshods
Both models show a similar increase
difference COSMO-2 minus COSMO-7 stays equal, resp. becomes littler for high time-windows
For Upscaling the influence of a time tolerance is relatively low und restricted on low thresholds ( effect of the avergaing)
 Application of time-windows on the gridsclae would make sense; simultaneous application with space tolerance brings no great change
T. Weusthoff

51. Precipitation [mm/12h]: frequency bias [in %] (1)
threshold
mod/domain
10S3
10S4
11S1
11S2
11S3
7_all 98
106
128
113 90
7_alps
112
125
157
143
0.1 mm
7_CH
124
145
168
169
2_alps
107
119
135
2_CH
120
136
148
150 85
102 93 87
126
149
122
1.0 mm
137
100
144
130
111
142
156
140

52. Precipitation [mm/12h]: frequency bias [in %] (2)
threshold
mod/domain
10S3
10S4
11S1
11S2
11S3
7_all 67 85 95 70 63
7_alps
115
122
100
5 mm
7_CH 91
138
141
2_alps 94
118
127
114
2_CH
103
143
135
123 62 80 72 55 57 93
10 mm 86
142 87
119
110
102
147
144
105
Compared to the previous year: differences > 15% only during Winter 09/10 for threshold 5 and 10 mm/12h over CH-domain for COSMO-7 and alps + CH-domain for COSMO-2 with > 15% less precipitation (especially over CH-domain for 10 mm/12h)

53. Precipitation [mm/12h]: frequency bias [in %] (3)
threshold
mod/domain
10S3
10S4
11S1
11S2
11S3
7_all 43 55 31 30 28
7_alps 54 67 34 38
30 mm
7_CH 78 86
(7) 35
2_alps
118 91 74 95
2_CH
130
108
(35) 85 87
113
131
107 97
104
120
136
124
146
147
126
Summary: mean bias

54. 4. Verifikation anderer Modellparameter  VERA
verfügbare VERA Parameter (nur für 2007): 10 m wind (u,v), pot. temperature, equiv. pot. temperature, mixing ratio, moisture flux convergence

55. „Mixing ratio“ im Vergleich zu VERA Analyse konsequent unterschätzt
„Neighbourhood“ Verifikation, COSMO-2, stündlich ab Leadtime +3h,
Jun-Aug 2007 (oben) und Sep – Nov 2007 (unten)
mixing ratio [g/kg]
„Mixing ratio“ im Vergleich zu VERA Analyse konsequent unterschätzt
zeitliche und räumliche Struktur recht gut erfasst

56. 4. Verifikation anderer Modellparameter  CM SAF (Globalstrahlung)
Globalstrahlung [W/m^2]
Differenz Globalstrahlung [W/m^2]
(model-obs)
Verifikation der Globalstrahlung aus COSMO-2 gegen CM-SAF
(Bsp. 1 Juli UTC)

57. „Neighbourhood“ Verifikation, COSMO-2, stündlich ab Leadtime +3h
Verifikation der Globalstrahlung aus COSMO-2 gegen CM-SAF
(Bsp. 1 Juli UTC)

58. Zusammenfassung (IV)
Anwendung der „Neighbourhood“ Verifikation auf andere Parameter prinzipiell möglich
vor allem für räumlich stark variable Daten
weitere Untersuchungen bzw. Anpassungen nötig, auch bezüglich der Beobachtungsdaten
z.B. Wolkenfilter für Strahlungsdaten