1. © Crown copyright Met Office WAFC CAT verification Objective verification of GRIB CAT forecasts Dr Philip G Gill, WAFS Workshop on the use and visualisation of gridded SIGWX forecasts, Paris, 14 September 2009

2. © Crown copyright Met Office Contents This presentation covers the following areas Introduction SIGWX forecast comparison Aircraft data Verification methodology Verification results Summary Further improvements

3. © Crown copyright Met Office Introduction What – Objective verification of gridded binary (GRIB) and significant weather (SIGWX) Clear Air Turbulence (CAT) forecasts Where – Global verification When – November 2008 to May 2009 Why – To demonstrate the quality of the new GRIB forecasts using objective verification. How – Verification against aircraft observations from the Global Aircraft Data Set (GADS)

4. © Crown copyright Met Office Manual SIGWX BUFR and GRIB CAT forecasts Manual SIGWX chart Example field from new GRIB forecast

5. © Crown copyright Met Office Comparison of SIGWX BUFR charts UKUSUK&US

6. © Crown copyright Met Office SIGWX BUFR CAT forecast comparison One month UK-US comparison (January 2009) Average coverage of globe UK ~6%, US ~3% Percentage overlap of all forecasts between UK and US ~20% Areas forecast by both UK and US Areas forecast by UK but not USAreas forecast by US but not UK

7. © Crown copyright Met Office GRIB forecast comparison produced by HKO

8. © Crown copyright Met Office Global Aircraft Data Set Archive of aircraft data set up by Joel Tenenbaum (State University of New York) British Airways fleet of Boeing 747-400 aircraft Global coverage, but flights mainly over northern hemisphere Automated aircraft observations every 4 seconds Indicator of turbulence derived from vertical acceleration, aircraft mass, altitude and airspeed called the derived equivalent vertical gust (DEVG).

9. © Crown copyright Met Office GADS Data coverage 10-day sample of GADS data

10. © Crown copyright Met Office Verification methodology

11. © Crown copyright Met Office Forecast assessment Turbulent/non turbulent event defined on 10min aircraft track ~120km - approx grid size Forecast turbulent event – CAT potential >= Threshold Observed turbulent event – Slight or greater turb (DEVG>=2m/s) Moderate or greater turb (DEVG>=4.5m/s) Construct 2x2 contingency tables for each threshold Sum entries in contingency tables over the verification period Produce a Relative Operating Characteristic (ROC) curve by plotting the Hit rate against False alarm rate for each threshold. Turbulence observed No turbulence observed Turbulence forecast HitFalse alarm No turbulence forecast MissCorrect rejection 2x2 contingency table

12. © Crown copyright Met Office Results UK GRIB and SIGWX moderate or greater turbulence ~500,000 events ~300 turbulent events (devg>=4.5m/s)

13. © Crown copyright Met Office Variation with forecast range ~100 000 events Devg>=2m/s

14. © Crown copyright Met Office Latitudinal variation UK GRIB moderate or greater turbulence 20S to 20N ~60,000 events ~100 turb events 90N to 50N ~200,000 events ~50 turb events 20N to 50N ~200,000 events ~200 turb events 50S to 20S ~12,000 events ~10 turb events

15. © Crown copyright Met Office Latitudinal variation US GRIB moderate or greater turbulence 20S to 20N ~30,000 events ~50 turb events 90N to 50N ~280,000 events ~200 turb events 20N to 50N ~100,000 events ~100 turb events 50S to 20S ~6,000 events ~3 turb events

16. © Crown copyright Met Office UK and US Nov 2008 and Jan 2009 ROC curve ~200 000 events Devg>=2m/s

17. © Crown copyright Met Office UK GRIB and SIGWX ~300 000 events Devg>=2m/s SIGWX automated object - UK automated SIGWX chart production system based on GRIB data

18. © Crown copyright Met Office Summary of results Both UK and US GRIB products show more skill than the manual SIGWX products. Global UK and US GRIB CAT forecasts score similarly Slight difference in scores as forecast range increases Some differences in scores at individual latitude bands – best performance between 20N and 90N. UK CAT coverage on manual BUFR SIGWX charts greater than the US

19. © Crown copyright Met Office Further improvements Automate verification process Improve consistency of forecasts by analysing verification data and altering production systems. Use verification to test future model upgrades and re- tune algorithms

20. © Crown copyright Met Office Questions and answers