1. By Nicholas Leonardo Advisor: Brian Colle
Verification of Multi-Model Ensemble Forecasts of North Atlantic Tropical Cyclones and an Analysis of Large Track Error Cases
By Nicholas Leonardo
Advisor: Brian Colle

2. Motivation
Evacuation models are heavily dependent on track forecasts made 3-5 days in advance.
On average, deterministic track forecasts have improved over the past decades.
There are no comprehensive verification of multi-model ensemble systems spanning several years.

3. Motivation
Isabel (2003)
Joaquin (2015)
Would need more time to find/make a plot for Mathew (2016) instead... It however seemed more like a >5day issue when it was crossing over the DR region; otherwise I have yet to see if it is a “large error” case (eg. which forecast(s) fit the criteria I’ve set)
Some storms are less predictable than others, resulting in large track and/or intensity errors.
WRF-ARW initialized 2003 Sept 15th 0000UTC. Every 6 hours plotted up to Sept 19th 1200UTC.

4. Outline Objectives Data and Methods Model Verification
Probabilistic Scores
Large Error Cases
Conclusions

5. Objectives
Verify 3 global ensembles and several deterministic models for North Atlantic hurricane tracks, focusing on 3-5 day forecasts over the past 8 years.
Determine whether the ensembles have probabilistic skill relative to the EC deterministic or the NHC uncertainty cones.
Identify the outlier storms in terms of track error.
Determine what common patterns might be contributing to these large error cases.
Do the sources of error tend to be local or remote relative to the TC?

6. Data and Methods
Forecast cyclone tracks from the ECMWF (51 members), GEFS (21 members), and UKMET (23 members) ensemble prediction systems ( and those from the deterministic CMC, ECdet, GFSdet, UKMdet, NGPS, GFDL, HWRF, and the NHC’s official (OFCL) forecasts (ftp://ftp.nhc.noaa.gov/atcf/archive/).
Verified with the NHC best track data for 109 named storms from 2008 to 2015.

7. 2008-2015 Mean Track Error vs. Lead Time
Model Verification
Homogenous comparison of track errors as a function of lead time.
Included the combination of the 3 ensemble prediction systems or grand ensemble (“GE”).
The ECMWF deterministic (“ECdet") does even better than the GE mean.
ECdet also has a higher percentage superiority among the models.
The grand ensemble is never the worse.
The high-resolution coupled HWRF struggles.
Lead Time (h)
Percentage Best Track Forecasts
Percentage Worst Track Forecasts
Lead Time (h)
Lead Time (h)

8. Model Verification
There appears to be a slow bias in all of the models.
The average cross track errors are smaller in magnitude, though the GEFS/GFSdet show a right-of-track bias at 120h
Along-Track Error vs. Lead Time
Cross-Track Error vs. Lead Time

9. Model Verification
Some models have a decrease in track error, especially
This corresponds to an overall improvement in the slow bias.
Annual Averages of 72h-120h Total Track Errors
Annual Averages of 72h-120h Along-Track Errors

10. Model Verification
Much of the slow bias can be attributed to extratropical transitions (“ET”; eg. Buckingham et al. 2010).
Excluding the verification of observed ET events reduces this mean bias in all of the models.
Mean Along-Track Error
Mean Along-Track Error (w/out ET)

11. Probabilistic Scores
Brier Skill Score vs. Threshold Distance
Evaluated the ensembles with the Brier Skill Score:
Summed all n h forecast probabilities ( f ) against observations (o) that a TC will come within a threshold distance from a point. Used the ECdet as the reference.
GE and ECMWF show the most skill. UKMET struggles for larger threshold distances. _

12. Probabilistic Scores BSS (96h-120h)
Cone radius
Used the OFCL (with climo-based cone) as the reference: a randomly-generated 1000-member ensemble with a Gaussian distribution of distances from the OFCL position in vector space. The standard deviation of these distances is specified by the cone radius.
Compared 96h/120h forecasts of the ensembles with OFCL; each forecast pair having the same initialization date.
GE and ECMWF show skill over the baseline for distances >200km.
UKMET tends to struggle the most.

13. Probabilistic Scores
BSS (72h-96h; ensemble interpolated)
Reliability Diagram (d < 500km)
Repeated the experiment, but comparing the 72h/96h OFCL forecast with the 84h/108h ensemble forecasts initialized 12h earlier (interpolated).
The GE and ECMWF are only comparable to the baseline.
The UKMET again has poorer skill.
Reliability diagrams imply GEFS and UKMET over-forecasts high hit probabilities.

14. Probabilistic Scores
Not terribly important; delete to save a minute?
Rank histograms show all ensembles are under-dispersed in the along-track direction, with a clear slow-bias.
ECMWF is slightly over-dispersed in the cross-track, while UKMET and GEFS are under-dispersed.

15. Large Error Cases Small Error Cases Large Error Cases
Nfcst = 30
MAE = 142km
Nfcst = 31
MAE = 707km
Sorted maximum track errors of the GE mean beyond 72h and took the top (bottom) 25% largest errors.
Only included two different forecasts for the same TC if they were separated by >5 days.
Initializations of many large error cases appear to be clustered north and east of Puerto Rico and associated with recurving TC’s
Most small error cases did not move far throughout the forecast and/or were mostly driven by steady easterlies

16. Large Error Cases
Used ensemble sensitivity analysis (ESA): correlating ensemble gridded fields with peak member track errors (eg. Chang et al., 2013).
Also analyzed standardized differences (eg. Torn et al. 2015).
Members were sorted by track errors. The mean field of the 10 “good” members is subtracted from that of the 10 “bad” and normalized by the full sample standard deviation
Tracked the uncertainty sources related to each large error case to earlier forecast hours. The cases were classified based on the initial proximity of the identified source(s) relative to the TC.
The Number of Large Track Error Cases With Uncertainty Source Types
Source Type
Number
Remote Upstream 3
Nearby Upstream 10
Nearby Downstream 9
TC-Local 14
Undefined

17. Large Error Cases Type 1: Remote Upstream Uncertainty:
Associated with some synoptic feature >5000km upstream from the TC (eg. wave-guide-like pattern over the eastern North Pacific) .
Ensemble Mean 300mb Heights and ESA with Max Track Error (024h)
Step through very quickly... Do so with the other 4 we discussed (>1 minute to explain each)

18. Large Error Cases Type 1: Remote Upstream Uncertainty:
Associated with some synoptic feature >5000km upstream from the TC (eg. wave-guide-like pattern over the eastern North Pacific) .
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (024h)
Step through very quickly... Do so with the other 4 we discussed (>1 minute to explain each)

19. Large Error Cases Type 1: Remote Upstream Uncertainty:
Associated with some synoptic feature >5000km upstream from the TC (eg. wave-guide-like pattern over the eastern North Pacific) .
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (048h)

20. Large Error Cases Type 1: Remote Upstream Uncertainty:
Associated with some synoptic feature >5000km upstream from the TC (eg. wave-guide-like pattern over the eastern North Pacific) .
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (060h)

21. Large Error Cases Type 1: Remote Upstream Uncertainty:
Associated with some synoptic feature >5000km upstream from the TC (eg. wave-guide-like pattern over the eastern North Pacific) .
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (072h)

22. Large Error Cases Type 1: Remote Upstream Uncertainty:
Associated with some synoptic feature >5000km upstream from the TC (eg. wave-guide-like pattern over the eastern North Pacific) .
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (096h)

23. Large Error Cases Type 2: Nearby Upstream Uncertainty:
Uncertainty attached to an upstream feature that directly steers the TC at a later time.
Ensemble Mean 300mb Heights and ESA with Max Track Error (024h)

24. Large Error Cases Type 2: Nearby Upstream Uncertainty:
Uncertainty attached to an upstream feature that directly steers the TC at a later time.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (024h)

25. Large Error Cases Type 2: Nearby Upstream Uncertainty:
Uncertainty attached to an upstream feature that directly steers the TC at a later time.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (048h)

26. Large Error Cases Type 2: Nearby Upstream Uncertainty:
Uncertainty attached to an upstream feature that directly steers the TC at a later time.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (072h)

27. Large Error Cases Type 2: Nearby Upstream Uncertainty:
Uncertainty attached to an upstream feature that directly steers the TC at a later time.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (096h)

28. Large Error Cases Type 3: Nearby Downstream Uncertainty:
Uncertainty attached to a downstream (east of the TC’s longitude) feature steering the TC.
Ensemble Mean 300mb Heights and ESA with Max Track Error (0h)

29. Large Error Cases Type 3: Nearby Downstream Uncertainty:
Uncertainty attached to a downstream (east of the TC’s longitude) feature steering the TC.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (036h)

30. Large Error Cases Type 3: Nearby Downstream Uncertainty:
Uncertainty attached to a downstream (east of the TC’s longitude) feature steering the TC.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (048h)

31. Large Error Cases Type 3: Nearby Downstream Uncertainty:
Uncertainty attached to a downstream (east of the TC’s longitude) feature steering the TC.
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (072h)

32. Large Error Cases Type 4: TC-related:
The signal first appears near the TC or in the immediate TC environment (maybe driven by the TC itself).
Ensemble Mean 300mb Heights and ESA with Max Track Error (036h)

33. Large Error Cases Type 4: TC-related:
The signal first appears near the TC or in the immediate TC environment (maybe driven by the TC itself).
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (048h)

34. Large Error Cases Type 4: TC-related:
The signal first appears near the TC or in the immediate TC environment (maybe driven by the TC itself).
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (072h)

35. Large Error Cases Type 4: TC-related:
The signal first appears near the TC or in the immediate TC environment (maybe driven by the TC itself).
Ensemble Mean 300mb Heights and GOOD10-BAD10 Difference (096h)

36. Conclusions
The ECMWF ensemble has among the smallest MAE’s for TC track.
All of the models have a slow bias in along-track at days 3-5, largely due to TC’s undergoing extratropical transitions.
Track forecasts have improved in HWRF and GEFS during the last several years, with GEFS MAE’s becoming more comparable to the ECMWF.
The ECMWF and GE have the best probabilistic forecasts. Their skill is better than the deterministic ECdet and comparable to the OFCL cone forecast (even after interpolating the ensemble).
The largest track error cases appear to be clustered north and east of Puerto Rico, associated with recurving TCs.
For many of these cases, the physical sources of uncertainty associated with the track can be traced back to processes occurring in the near-TC environment and to the phase and amplitude of upstream steering trough/ridge systems at earlier lead times.

37. References
Buckingham, C., T. Marchok, I. Ginis, L. Rothstein, and D. Rowe, 2010: Short- and Medium-Range Prediction of Tropical and Transitioning Cyclone Tracks within the NCEP Global Ensemble Forecasting System. Weather and Forecasting, 25, 1741–1742.
Chang, E. K. M., M. Zheng, and K. Raeder, 2013: Medium-Range Ensemble Sensitivity Analysis of Two Extreme Pacific Extratropical Cyclones. Mon. Wea. Rev., 141, 211–231, doi: /MWR-D
Torn, R. D., J. S. Whitaker, P. Pegion, T. M. Hamill, and G. J. Hakim, 2015: Diagnosis of the Source of GFS Medium-Range Track Errors in Hurricane Sandy (2012). Mon. Wea. Rev., 143, 132–152, doi: /MWR-D