1. © Crown copyright Met Office Preliminary results using the Fractions Skill Score: SP2005 and fake cases Marion Mittermaier and Nigel Roberts

2. © Crown copyright Met Office Outline 1.What is a good forecast? 2.Introduction to the method 3.A real case: 31 May 2005 “As is” results Domain size influences 4.Synthetic cases Perturbed 31 May 2005 cases Geometric 5.Conclusions

3. © Crown copyright Met Office What makes a good forecast?

4. © Crown copyright Met Office What is a good rainfall forecast?

5. © Crown copyright Met Office Background Higher-resolution precipitation forecasts look more realistic, but are computationally expensive. Are they more accurate? Traditional point verification is inappropriate. We shouldn’t believe the small scales. Can a ~1-km model provide more accurate rainfall forecasts on useful scales (e.g. over river catchments)? How should we interpret and present the model output.

6. © Crown copyright Met Office 1.We can never get the initial conditions exactly right (worse at higher resolution) and forecast errors grow 2.Finer resolution introduces faster growing errors at the smaller scales. Small scales become unpredictable quickly. (Lorenz 1969, Zhang et al 2003) Each forecast is one possible solution from a pdf of alternatives We must use probabilities (or other indicators of uncertainty) to deal with unpredictable scales when presenting high-resolution precipitation forecasts. But what scales are unpredictable? High-resolution model predictability

7. © Crown copyright Met Office The method

8. © Crown copyright Met Office Theis et al 2005 RUC model - Weygandt and Benjamin, 2005 Nimrod nowcasting system, UK Met Office Probabilities come from fractions of occurrences within neighbourhoods What neighbourhood size should we use? Probabilities from nearest neighbours

9. © Crown copyright Met Office 3 rd August 2004 1-km model forecast of 6-hour rainfall accumulations Averaged to 5-km grid Select accumulations exceeding the 95 th percentile value ~ 20mm Binary probability 1 or 0 Probabilities/fractions from a square neighbourhood of length 35 km Still quite sharp Useful if forecast is accurate Probabilities/fractions from a square neighbourhood of length 65 km More appropriate for a less accurate forecast system Smoother – loss of sharpness Probabilities/fractions from a square neighbourhood of length 125 km More appropriate for a much less accurate forecast system. Much smoother – this would not be a cost effective use of a 1-km model. Example of neighbourhood probabilities/fractions

10. © Crown copyright Met Office Verification approach We want to know: 1.How forecast skill varies with neighbourhood size. 2.The smallest neighbourhood size that can be used to give sufficiently accurate forecasts. 3.Does higher resolution provide more accurate forecasts on scales of interest (e.g. river catchments) Compare forecast fractions with fractions from radar over different sized neighbourhoods (squares for convenience) Use rainfall accumulations to apply temporal smoothing

11. © Crown copyright Met Office Schematic comparison of fractions Threshold exceeded where squares are blue observedforecast

12. © Crown copyright Met Office Brier score for comparing fractions A score for comparing fractions with fractions Skill score for fractions/probabilities - Fractions Skill Score (FSS)

13. © Crown copyright Met Office Example graph of FSS against neighbourhood size

14. © Crown copyright Met Office Idealised example

15. © Crown copyright Met Office In summary This verification method provides a way of answering some important questions about forecasts from ‘storm-resolving’ NWP models. How does forecast skill vary with spatial scale? At what scales are higher resolution forecasts more skilful (if any)? At what scales are forecasts sufficiently accurate?........... (There are other questions that need different approaches)

16. © Crown copyright Met Office How we are using it 10% threshold FSS L(FSS>0.5) Freq. Bias 12.5 mm 0.5 20 km

17. © Crown copyright Met Office Case 31 May 2005

18. © Crown copyright Met Office CAPS - truthNCEP - truth NCAR - truth THE WINNER

19. © Crown copyright Met Office With such a large domain the wet-area ratio is quite small, even for extensive precipitation areas. How sensitive are the results to domain size? Perform tests where around ~30% of the domain is used Impact of domain size Area 0 1 2 3 45

20. © Crown copyright Met Office NCEP - truth 90 km 85 km 100 km 30 km 60 km > 200 km

21. © Crown copyright Met Office Points to ponder… Domain size affects the magnitude of the FSS and the spatial scales. Smaller domains are faster to compute. Sub-regions take into account that neither precipitation nor skill is uniform over a large domain. Without becoming object-oriented, an objective (and intelligent) selection procedure for identifying smaller sub-regions for verification may be very useful.

22. © Crown copyright Met Office Perturbed forecasts

23. © Crown copyright Met Office 001 - truth003 - truth 002 - truth 20 km 80 km 40 km 3pts R 5 pts dwn 6 pts R 10 pts dwn 12 pts R 20 pts dwn THE WINNER

24. © Crown copyright Met Office 004 - truth006 - truth 005 - truth 50 km 80 km 160 km 24 pts R 40 pts dwn 48 pts R 80 pts dwn 12 pts R 20 pts dwn, x1.5

25. © Crown copyright Met Office 007 - truth 35 km Case> 0.5 mm> 32 mm 0014 km20 km 0024 km40 km 00310 km80 km 00450 km190 km 005160 km> 200 km 00610 km? 00735 km90 km Summary : L(FSS > 0.5) 12 pts R 20 pts dwn, - 0.05

26. © Crown copyright Met Office Conclusions: perturbed cases Small shifts/timing errors are initially unharmful to the FSS and scale. Gross timing errors result in large degradations in skillful scales and the overall magnitude of the FSS. Changes in the actual magnitudes (006 and 007) show greater impact. Subtle differences (due to near-threshold level misses) seem to be potentially more serious in terms of FSS magnitude.

27. © Crown copyright Met Office Blobs

28. © Crown copyright Met Office 001 - truth003 - truth 002 - truth > 600 km 200 km THE WINNER

29. © Crown copyright Met Office 004 - truth 005 - truth 550 km ? Case> 10 mm 001200 km 002> 600 km 003> 600 km 004550 km 005? Summary : L(FSS > 0.5)

30. © Crown copyright Met Office Conclusions: geometric cases Case 005 has the largest FSS at the grid scale but in terms of L(FSS > 0.5) never reaches a skillful scale. Case 001 is the most skillful in terms of scale but has no skill at all at the grid scale, based on the FSS.

31. © Crown copyright Met Office General conclusions The absolute value of FSS is less useful than the scale where an acceptable level of skill is reached. The skillful scales in the fake cases ties in well with the 1-d idealized example results. Frequency thresholds are potentially useful but not when the domains are very large. [The zeros dominate the cdf.] Consider a “roving” (fixed size) verification domain that focuses on an area of interest. Other optimizing ideas are being explored.

32. © Crown copyright Met Office Questions and answers