1. Verification of Tropical Cyclone Forecasts
Beth Ebert (BOM)
Barb Brown (NCAR)
Laurie Wilson (RPN)
Tony Eckel (ERT)
8th TIGGE Working Group Meeting
22-24 February 2010, Geneva

2. What kind of TC forecasts?
Deterministic
TC track
Intensity
maximum wind
central pressure
temporal trend (rapid intensification)
Wind field / radii
Precipitation
Storm surge
Temporal consistency
Ensemble
Track distribution
strike prob., cone of uncertainty
Intensity distribution
mean / median
spread
90th percentile
Prob (wind > threshold)
Prob (precip > threshold)
Prob (surge > threshold)
Temporal consistency

3. Different users need different kinds of verification information
Public and emergency managers
Simple, graphical
Focus on impact
Forecasters
Information on how to interpret forecasts
Timing errors
Modellers
Systematic errors
How to improve the model
How to improve ensemble distribution / spread

4. Quality of deterministic forecasts
What are the track errors (along-track, cross-track)?
What are the intensity errors?
Are temporal intensity trends correctly predicted?
What is the error in timing of landfall?
What is the error in forecast maximum wind (rain)?
Multi-day total precipitation
Is the spatial distribution of wind (rain) correct?

5. Quality of ensemble forecasts
Does the ensemble enclose the observed track?
Are the ensemble probabilities skillful and reliable predictions for
strike probability (track)
intensity (max wind, central pressure)
wind
precipitation
Does the ensemble produce an appropriate spread for these variables?

6. Data issues Verification data Forecast data Reference forecasts
Data sources
Best track, Dvorak, surface instruments, radar, …
Problems measuring in extreme conditions
Forecast data
Size of ensemble
Model grid resolution
Details of cyclone tracker
Reference forecasts
Statistical forecast (e.g., CLIPER)

7. Verification methods for deterministic TC forecasts
Example 1 – visual comparison
Analysis
Model
Track/intensity verification
Courtesy Noel Davidson, BOM

8. Verification methods for deterministic TC forecasts
Example 2 – along-track and cross-track errors
Courtesy James Franklin, NHC

9. Verification methods for deterministic TC forecasts
Example 3 – cumulative distribution of track errors
Courtesy James Franklin, NHC

10. Verification methods for deterministic TC forecasts
Example 4 – distribution of intensity errors
HFIP High-Resolution Hurricane Test – DTC Final Report Sept 2009

11. Verification methods for deterministic TC forecasts
Example 5 – rapid intensification
Count
Hours since onset of observed RI
Observed
High resolution model
Low resolution model
AOML / WRF – 69 cases
HFIP High-Resolution Hurricane Test – DTC Final Report Sept 2009

12. Verification methods for deterministic TC forecasts
Example 6 – rain intensity distribution
Marchok et al. 2008

13. Verification methods for ensemble TC forecasts
Example 1 – visual comparison

14. Verification methods for ensemble TC forecasts
Example 2 – probabilistic scores and methods
MOGREPS 120 h forecast for
strike probability (within 120 km)
TC Anja, November 2009
fake ^

15. Verification methods for ensemble TC forecasts
Example 3 – spread of track and intensity forecasts
track
track
intensity
intensity
20-member FIM ensemble

16. New approaches for verifying TCs
Spatial verification methods
Precipitation and
wind fields
Storm characteristics
location
size
intensity
shape, etc.

17. New approaches for verifying ensemble TC forecasts
Minimum spanning tree
multi-variate rank histogram
Ensemble of object properties
ensemble mean object properties
distributions – use standard methods for ensembles and probability forecasts
relationship of TC genesis to the number of ensemble members predicting the TC at day 1+?
correspondence ratio
observation
ensemble
forecast
ensemble
forecast

18. Reporting guidelines Provide all relevant information
Model(s), grid, range of dates, forecast lead times, verification data source, etc.
Aggregation and distribution of results
Confidence intervals / uncertainty

19. Verification tools for TCs
US Developmental Testbed Centre (DTC)
Tracker (Marchok)
Code for verification of TC track and intensity
Other international tool sets??

20. Document on TC verification – commented literature review to be written this year
Contents
Introduction
Verification strategy
Reference data
Verification methods
Reporting guidelines
Summary
References
Appendices:
a. Brief description of scores
b. Guidelines for computing aggregate statistics
c. Confidence intervals for verification scores
d. Examples of graphical verification products