1. NCAR, 15 April 2008 1 Fuzzy verification of fake cases Beth Ebert Center for Australian Weather and Climate Research Bureau of Meteorology

2. NCAR, 15 April 2008 2 Look in a space / time neighborhood around the point of interest –Evaluate using categorical, continuous, probabilistic scores / methods –Will only consider spatial neighborhood for fake cases Fuzzy (neighborhood) verification t t + 1 t - 1 Forecast value Frequency Forecast value Frequency forecast observation

3. NCAR, 15 April 2008 3 Fuzzy verification framework Fuzzy methods use one of two approaches to compare forecasts and observations: single observation – neighborhood forecast (user-oriented) neighborhood observation – neighborhood forecast (model-oriented) observationforecast observationforecast

4. NCAR, 15 April 2008 4 Fuzzy verification framework good performance poor performance

5. NCAR, 15 April 2008 5 Upscaling Neighborhood observation - neighborhood forecast Average the forecast and observations to successively larger grid resolutions, then verify as usual % change in ETS Weygandt et al. (2004)

6. NCAR, 15 April 2008 6 Fractions skill score Neighborhood observation - neighborhood forecast observedforecast Compare forecast fractions with observed fractions (radar) in a probabilistic way over different sized neighbourhoods Roberts and Lean (2008)

7. NCAR, 15 April 2008 7 single threshold ROC Spatial multi-event contingency table Single observation - neighborhood forecast Vary decision thresholds: magnitude (ex: 1 mm h -1 to 20 mm h -1 ) distance from point of interest (ex: within 10 km,...., within 100 km) timing (ex: within 1 h,..., within 12 h) anything else that may be important in interpreting the forecast Fuzzy methodology – compute Hanssen and Kuipers score HK = POD – POFD Measure how close the forecast is to the place / time / magnitude of interest. Atger (2001)

8. NCAR, 15 April 2008 8 Practically perfect hindcasts Single observation - neighborhood forecast Q: If the forecaster had all of the observations in advance, what would the "practically perfect" forecast look like? –Apply a smoothing function to the observations to get probability contours, choose yes/no threshold that maximizes CSI when verified against obs –Did the actual forecast look like the practically perfect forecast? –How did the performance of the actual forecast compare to the performance of the practically perfect forecast? Fuzzy methodology – compute forecast PracPerf CSI forecast = 0.34CSI PracPerf = 0.48 Kay and Brooks (2000)

9. NCAR, 15 April 2008 9 1 st geometric case 50 pts to the right bad good 12.7 mm 25.4 mm

10. NCAR, 15 April 2008 10 2 nd geometric case 200 pts to the right bad good

11. NCAR, 15 April 2008 11 5 th geometric case 125 pts to the right and huge bad good

12. NCAR, 15 April 2008 12 1 st case vs. 5 th case ~same Case 1 better Case 5 better

13. NCAR, 15 April 2008 13 Perturbed cases 1000 km "Observed" (6) Shift 12 pts right, 20 pts down, intensity*1.5 (4) Shift 24 pts right, 40 pts down Which forecast is better?

14. NCAR, 15 April 2008 14 4 th perturbed case 24 pts right, 40 pts down bad good

15. NCAR, 15 April 2008 15 6 th perturbed case 12 pts right, 20 pts down, intensity*1.5 bad good

16. NCAR, 15 April 2008 16 Difference between cases 6 and 4 Case 4 - Shift 24 pts right, 40 pts down Case 6 - Shift 12 pts right, 20 pts down, intensity*1.5 Case 6 – Case 4 6 4

17. NCAR, 15 April 2008 17 How do fuzzy results for shift + amplification compare to results for the case of shifting only? Case 6 - Shift 12 pts right, 20 pts down, intensity*1.5 Case 3 - Shift 12 pts right, 20 pts down, no intensity change Case 6 – Case 3 3 Why does the case with incorrect amplitude sometimes perform better?? Baldwin and Kain (2005): When the forecast is offset from the observations most scores can be improved by overestimating rain area, provided rain is less common than "no rain". 6

18. NCAR, 15 April 2008 18 Some observations about methods Traditional Measures direct correspondence of forecast and observed values at grid scale Hard to score well unless forecast is ~perfect Requires overlap of forecasts and obs Entity-based (CRA) Measures location error and properties of blobs (size, mean/max intensity, etc.) Scores well if forecast looks similar to observations Does not require much overlap to score well Fuzzy Measures scale- and intensity-dependent similarity of forecast to observations Forecast can score well at some scales and not at others Does not require overlap to score well

19. NCAR, 15 April 2008 19 Some final thoughts… Object-based and fuzzy verification seem to have different aims Object-based methods Focus on describing the error What is the error in this forecast? What is the cause of this error (wrong location, wrong size, wrong intensity, etc.)? Fuzzy neighborhood methods Focus on skill quantification What is the forecast skill at small scales? Large scales? Low/high intensities? What scales and intensities have reasonable skill? Different fuzzy methods emphasize different aspects of skill

20. NCAR, 15 April 2008 20 Some final thoughts… When can each type of method be used? Object-based methods When rain blobs are well defined (organized systems, longer rain accumulations) When it is important to measure how well the forecast predicts the properties of systems When size of domain >> size of rain systems Fuzzy neighborhood methods Whenever high density observations are available over a reasonable domain When knowing scale- and intensity-dependent skill is important When comparing forecasts at different resolutions