1. Page 1© Crown copyright Scale selective verification of precipitation forecasts Nigel Roberts and Humphrey Lean

2. Page 2© Crown copyright Background - What do we want to know? 1km model – should improve precipitation forecasts …. But, smaller scales are less predictable (Zhang et al 2003) – yet higher expectation Can a 1-km model provide more accurate and useful forecasts of rainfall events on the scales of river catchments? On what scales should the output be presented? What does data assimilation do to different scales?

3. Page 3© Crown copyright Verification approach Verify over different spatial scales using a conceptually simple approach. Fractions/probabilities from nearest neighbouring points. (also Ebert) Verify against radar – good spatial coverage. Stable network over UK. Verify accumulations - smooth out temporal noise. Use accumulation exceedance thresholds e.g. > 4 mm, > 8 mm ….

4. Page 4© Crown copyright Radar12 km forecast1 km forecast 0.125 0.5 1 2 4 8 16 32 mm The problem we face 0100 km Six hour accumulations 10 to 16 UTC 13th May 2003

5. Page 5© Crown copyright Schematic example - different scales

6. Page 6© Crown copyright 1-km forecastRadar 0.125 0.5 1 2 4 8 16 32 mm Six hour accumulations 10 to 16 UTC 13th May 2003

7. Page 7© Crown copyright 4 mm threshold, Fractions at grid scale (1 or 0) ModelRadar > 4 mm 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fraction

8. Page 8© Crown copyright ModelRadar > 4 mm 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fraction 4 mm threshold, Fractions within 35x35 km squares

9. Page 9© Crown copyright ModelRadar 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fraction 4 mm threshold, Fractions within 75x75 km squares

10. Page 10© Crown copyright ModelRadar 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Fraction 4 mm threshold, Fractions within 105x105 km squares

11. Page 11© Crown copyright Brier score for comparing fractions Skill score for fractions/probabilities - Fractions Skill Score (FSS) A score for comparing fractions with fractions

12. Page 12© Crown copyright Graphical behaviour of the Fractions Skill Score

13. Page 13© Crown copyright Evaluation of precipitation forecasts from a 1km NWP forecast model 11 convective cases (Summer 2003/2004) 44 forecasts out to T+7 12km with data assimilation (3-hour cycle) Concentrating on 1km spun up from 12km fields at T+1 (no additional data assimilation at 1km)

14. Page 14© Crown copyright Results for 6-hour accumulations

15. Page 15© Crown copyright Results for 6-hour accumulations

16. Page 16© Crown copyright Results for 1-hour accumulations

17. Page 17© Crown copyright Scale selective evaluation of operational 12km model precipitation forecasts during 2003 hourly accumulations

18. Page 18© Crown copyright Conclusions about the approach  Verification approach is useful  the concept is intuitively easy to understand  gives information about spatial accuracy  links output products to verification  agreement with subjective assessment  Issues ….  choice and interpretation of scores, dealing with a bias  radar is regarded as ‘truth’!  variability from case to case  should be compared with other approaches – e.g. Barbara Casati

19. Page 19© Crown copyright Conclusions about the results  Forecasts of the spatial distribution of rainfall are more accurate at 1km than 12km at scales > ~20-30km.  The scale at which satisfactory skill is achieved is improved at 1km. e.g. from 70km (12km model) to 55km (1km model) for a 10% threshold  The more isolated the events being predicted the poorer the forecast skill, however the 1km model gave the greatest improvement for the most isolated events.  The impact of resolution is greater for hourly than for 6-hourly accumulations. Skill at 1km is maintained after the spin-up period.  Operational 12km model - Skill drops off most quickly after the analysis time at the smallest scales (~ <100 km).

20. Page 20© Crown copyright Caution - bias

21. Page 21© Crown copyright Scale selective evaluation of operational 12km model precipitation forecasts during 2003 widespread localised

22. Page 22© Crown copyright Variation between forecasts

23. Page 23© Crown copyright Scale selective evaluation of operational 12km model precipitation forecasts during 2003