Forecast skill Contingency tables can be used for various probability thresholds to produce a reliability table From the reliability table various scores can be calculated The skill of the forecast at discriminating between events and non-events can be measured using the Receiver Operating Characteristic (ROC) by plotting hit rate against false alarm rate The curve from a probabilistic forecast and single point from a deterministic forecast can be compared. Line of no skillPerfect forecast Area under curve measures skill
Latest results – mean CAT Acceptable hit rates? Acceptable false alarm rates? Area of interest could be measured by partial area under ROC curve
Latest results – max CAT
Latitudinal variation 50N to 90N 20S to 20N 20N to 50N 50S to 20S Good performance from above 20N
Summary of CAT verification Blending process continues to produce both consistent and skilful turbulence forecasts. Some variation in skill by latitude with best performance in the mid latitudes. Rolling 12-month results produced monthly since June 2012.
Cb verification Verification against lightning strikes (SFERICS) from Met Office ATDnet Gridded SFERICS field produced using observations at +/- 30 minutes from forecast validity time T+24 forecast verified from all four model runs
Latitudinal variation 50N to 80N20N to 50N 20S to 20N40S to 20S
Summary of Cb verification Blending process continues to produce both consistent and skilful Cb forecasts. Some variation in skill by latitude Rolling 12-month results produced quarterly since June 2012.
Improving turbulence verification Switch from DEVG to EDR ICAO standard Forecasting and observing EDR Source more EDR observations from other airlines to increase the area of coverage Increase coverage to fill gaps in current GADS database
Improving Cb verification Extend verification against lightning reports by outside of the ATDnet domain Source additional lightning reports outside [80N, 40S, 100W, 80E] Use geostationary satellite data in addition to lightning data MSG [60N, 60S] Overshooting convection product (work underway) New satellites over America (GOES-R) and East Asia (Himawari-8) Verify Cb cloud top height and Cb extent
MSG - the European Geostationary Imager Scans Africa, Europe, Middle East and Atlantic every 15 minutes 3/1 km spatial resolution at 0° N, 0° E 12 channels (different wavelengths) Meteosat-10 operational, Meteosat-8,-9 backup
Severe convection product Data from Meteosat Second Generation Calculated from the visible and infrared channels. Generated every 15 mins. Current product over Europe identifies area where severe convection is occurring (day-time only) New overshooting convection product will identify intense convective areas with overshooting cloud tops. Could be merged with lightning data from ATDnet Could be merged later with Meteosat third generation lightning data
Cloud top height product Data from Meteosat Second Generation Calculated from the infrared channels at wavelengths of 10.8, 12.0 and 13.4 microns. Generated every 15 mins Covers Europe, Atlantic, Africa Extend to global coverage following GOES-R and Himawari-8 satellite launches (~ 2015) Mask with Cb product
Verification – all forecasts Verify probabilistic forecasts using suitable metrics Forecast skill – ROC (partial area under curve?) Reliability diagram Value – relative economic value measure Publish verification results – plots and contingency tables Breakdown into WMO regions Forecast ranges T+12, T+24, T+36 Rolling 12-month scores Geostationary satellite data limited to 60N, 60S. Polar orbiter data could supplement this if necessary.
Timescales 2013/14 Preliminary work on satellite data specifications Routine CAT and Cb verification published by WMO region. 2014/15 Include satellite data in Cb verification Develop Cb cloud top height verification Move to using EDR in turbulence verification 2015/16 Expand coverage of Cb verification using new satellite data over East Asia (Himawari-8) 2016/17 Expand coverage of Cb verification using new satellite data over America (GOES-R)