1. Tropical Cyclone Verification Operations in STI/CMA,
Tropical Cyclone Verification Operations in STI/CMA
Hui YU
Shanghai Typhoon Institute
MEETING OF THE CBS (DPFS) EXPERT TEAM ON OWFPS
BEIJING, CHINA MARCH 2018

2. Outline History of TC forecast verification at STI/CMA
National level activities in China
International activities
Capability and commitment to fulfil the role of LC-TCFV as required in the Manual on GDPFS ( and )
Future plan

3. History of tropical cyclone forecast verification at STI (National level activities)
Tropical cyclone forecast data exchange around China
STI started the TC forecast evaluation authorized by CMA
Add landfall forecast evaluation
1972
1992
2009
2012
2014
2016
1985
2006
2010
2013
2015
Started to assess the performance of TC forecast
Track & intensity error analysis

4. History of tropical cyclone forecast verification at STI (National level activities)
Authoritative verification organization
Submit annual verification report to the National Working Group of Experts on Typhoon and Marine Meteorology.
Evaluate the prediction skill of all the forecast methods
Provide the recommendations regarding whether the methods could continue to be used in operation.
Rule: forecast guidance which get negative skill score for the latest two years will be disqualified.
TC Forecast Guidance in China
Subjective method
NWP models
EPS
Other objective method
CMA
T213
CMA-GEFS
GX-multi-fault
Shanghai
T639
STI-GRAPES
JS-probabilistic
Zhejiang
WIPS
Fujian
STI-WARMS
SH-XSLD
Guangdong
GZ-TRAMS
SH-PLSC
Hainan
LN-WRF
Jiangsu

5. History of tropical cyclone forecast verification at STI (International activities)
Metrics development:
TC genesis
Ensemble probability
Prediction system error
NWP characteristic analysis
(Supported by WMO-TLFDP coordination with JWGFVR)
Tropical cyclone forecast data exchange around China
STI started the TC forecast evaluation authorized by CMA
Add landfall forecast evaluation
1972
1992
2009
2012
2014
2016
1985
2006
2010
2013
2015
Started to assess the performance of TC forecast
Track & intensity error analysis
Built real-time & post-season verification systems (supported by WMO-TLFDP, coordination with JWGFVR)

6. Development of verification systems
History of tropical cyclone forecast verification at STI (International activities)
Development of verification systems
(Supported by WMO-TLFDP, coordination with JWGFVR)
Real-time verification system
Provide track, intensity, speed, direction error
Via website
Post-season verification system
Provide basic forecast errors
Implement new verification techniques
Comprehensive analysis of errors
Real-time verification web-page

7. (Supported by WMO-TLFDP, coordination with JWGFVR)
History of tropical cyclone forecast verification at STI (International activities)
Metrics development
(Supported by WMO-TLFDP, coordination with JWGFVR)
Schematic charts for TFID
Brier score of the seven TIGGE ensemble prediction systems in 2010

8. History of tropical cyclone forecast verification at STI (International activities)

9. The operational status of TC forecast verification up to 2012 is analyzed on the basis of an survey covering all the Members of UNESCAP/WMO Typhoon Committee
Main conclusions
Significant efforts have been focused on the verification of TC forecast guidance by operational forecast agencies in the region
Only a few verification products are available for the probabilistic forecasts from EPS
Verification of TC precipitation and high wind forecasts are also lacking sufficient attention in the region
CMA – China; DOM – Cambodia; DMHL – Laos; JTWC – USA; MMGB – Macao; NHMSV – Vietnam; NTC/KMA – Korea; PAGASA – Philippines
Yu, Chan, Brown, et al., 2012, TCRR
Invited talk at the 6th International Verification Methods Workshop in March 2014.

10. (Research fellowship projects of UNESCAP/WMO Typhoon Committee)
History of tropical cyclone forecast verification at STI (International activities)
Metrics development
(Research fellowship projects of UNESCAP/WMO Typhoon Committee)
Ms. Lethihai YEN
from Viet Nam (2013)
Mr. Boothum TANGLUMLEAD
from Thailand (2014)
Seven research fellowship projects were implemented as jointly supported by the UNESCAP/WMO Typhoon Committee, STI and HKO. The successful applicants of these projects visited STI or HKO for one or two months and implemented research.
Research topics:
TC genesis forecast evaluation technique
TC precipitation forecast evaluation technique
TC track ensemble forecast evaluation technique
Improvement of TC verification system
Mr. Yong Chol SONG
from DPRK (2014)
Mr. Sang Il PAK
from DPRK (2014, 2015,2016)

11. History of tropical cyclone forecast verification at STI (International activities)
Metrics development
A number of comments were provided to the authors of the WMO document “Verification
of tropical cyclone forecasts” (by JWGFVR), with special attention given to increase the
visibility of related efforts in the western North Pacific region. This document has been
published by the WMO and is available on the website of the WWRP.

12. History of tropical cyclone forecast verification at STI (International activities)
Metrics development:
TC genesis
Ensemble probability
Prediction system error
NWP characteristic analysis
(under WMO)
Tropical cyclone forecast data exchange around China
STI started the TC forecast evaluation authorized by CMA
Add landfall forecast evaluation
1972
1992
2009
2012
2014
2016
1985
2006
2010
2013
2015
Started to assess the performance of TC forecast
Track & intensity error analysis
Built real-time & post-season verification systems supported by WMO-TLFDP
Submit annual verification report to ESCAP/WMO Typhoon Committee
Research fellowship projects on verification
Typhoon Committee members: Vietnam, DPR Korea, Thailand

13. UNESCAP/WMO Typhoon Committee http://www.typhooncommittee.org
Established in 1968
Members：14
Cambodia, 1968 China, 1968 DPRK, 1992 HongKong/China, 1968 Japan, 1968 Lao PDR, 1968 Macao/China, 1992
Malaysia 1968 Philippines, 1968 RoK, 1968 Singapore, 1997 Thailand, 1968 USA, 1998 Viet Nam, 1968
An intergovernmental body in order to promote and coordinate the planning and implementation of measures required for minimizing the loss of life and material damage caused by typhoons in Economic and Social Commission for Asia and the Pacific region.

14. History of tropical cyclone forecast verification at STI (International activities)
Submit the annual TC verification report in western North Pacific region to Typhoon Committee since (Typhoon Committee PPP project).
The report has become the important technical material to determine the annual working plan of Typhoon committee.

15. History of tropical cyclone forecast verification at STI
Metrics development:
TC genesis
Ensemble probability
Prediction system error
NWP characteristic analysis
(under WMO)
Tropical cyclone forecast data exchange around China
STI started the TC forecast evaluation authorized by CMA
Add landfall forecast evaluation
Application for Lead Centre for Coordination of TCFV
1972
1992
2009
2012
2014
2016
1985
2006
2010
2013
2015
Started to assess the performance of TC forecast
Track & intensity error analysis
Built real-time & post-season verification systems supported by WMO-TLFDP
Submit annual verification report to ESCAP/WMO Typhoon Committee
Research fellowship projects on verification
Typhoon Committee members: Vietnam, DPR Korea, Thailand

16. Capability and commitment to fulfil the role of LC-TCFV as required in the Manual on GDPFS ( and )

17. Website Verification reports and related publications & presentations
Real time verification (charts and data)
WMO-TLFDP official website：

18. Parameter(s) MSLP (Mandatory) Maximum wind speed Track Precipitation
Note: TC track and intensity forecasts are provided by the producing centres on the GTS or FTP site

19. Verification datasets
Impacts of different best track data on the verification results are clear
Variation intervals of track errors for official guidances, global models and regional models by referring to different TC best tracks in 2017.

20. Scores
Detection rate (Not available. A score related to the vortex location technique.)
Basic verification
Statistical verification
Notes
Track
Deterministic
Position error
Along-track/cross-track (ATCT) bias
Direction error
Speed error
Track forecast integrated deviation (TFID)
Box-plots with SS information
Observed vs Forecast relative position images
Track error rose
Probability distribution analyses
Binned by : seasonal, regional, intensity, life-stage, steering flow
Probabilistic (EPS)
Hitting ratio of the probabilities
Bidirectional track forecast scatter plot for EPSs
Intensity
Bias
Absolute error
Tendency error
Category contingency table
Category-score
POD-FAR joint chart
Observed vs Forecast intensity images
Binned by : seasonal, regional, intensity, intensity change rate, life-stage
NWP initial error to be provided as an indicator of reliability
EPS history ranking
EPS BS
EPS PRS
Development of a baseline : climatology probabilistic forecast
Landfall
Timing error
Precipitation
Gridded
MET
Only for WARMS now
Need an expansion to more models
Stations
TS score, POD, FAR, Bias
Winds
High winds score, POD, FAR

21. Future Plans
STI is planning to further strengthen its capability to fulfil the role of LC-TCFV as required in the Manual.
Further develop the verification metrics as supported by WMO-TLFDP and in coordination with JWGFVR. A workshop is being planned in June 2018.
Extend the verification areas from NWP to other basins.
Provide the scores through the website in the required format.
Implement a standard vortex location technique for gridded outputs and carry out verification.
(GFDL vortex location technique has been implemented in STI for ECMWF-IFS, NCEP-GFS, and CMA-T639)

22. 谢 谢 ! Thank you for your attention !
谢 谢 !
MEETING OF THE CBS (DPFS) EXPERT TEAM ON OWFPS
BEIJING, CHINA MARCH 2018

23. Verifications for 2017
There has been a trend of decrease on model TC track forecast errors since 2010, with a small perturbation over last two years (2016 and 2017)
Box plots of position errors for ECMWF-IFS, NCEP-GFS and UKMO-MetUM in TC track forecasts from 2010 to The bar in the middle of the plot represents the median values of errors, the lower and upper ends of the boxes represent the 25th and 75th quantile values. The bars below and above the box represent the non-outlier extreme values, and the circles represent the outliers.

24. Verifications for 2017
Impacts of different best track data on the verification results are clear
Variation intervals of track errors for official guidances, global models and regional models by referring to different TC best tracks in 2017.

25. Over-estimated uncertainty Under-estimated uncertainty
Verifications for 2017
Over-estimated uncertainty
Under-estimated uncertainty
Bidirectional track forecast scatter plot for EPSs in The blocks represent the mean values of spread or position error. The lower (left) and upper (right) bars represent the 25th and 75th quantile values.

26. Verifications for 2017
There has been little progress on the performance of intensity prediction for both global and regional models since 2010
Intensity forecast skill scores at the lead times of 48 h for global and regional models.
Brier scores for EPS intensity forecast in 2017.