1. Forecast Verification Research Beth Ebert and Laurie Wilson, JWGFVR co-chairs WWRP-JSC meeting, Geneva, 21-24 Feb 2011

2. 2 Aims Verification component of WWRP, in collaboration with WGNE, WCRP, CBS Develop and promote new verification methods Training on verification methodologies Ensure forecast verification is relevant to users Encourage sharing of observational data Promote importance of verification as a vital part of experiments Promote collaboration among verification scientists, model developers and forecast providers

3. 3 Working group members  Beth Ebert (BOM, Australia)  Laurie Wilson (CMC, Canada) Barb Brown (NCAR, USA) Barbara Casati (Ouranos, Canada) Caio Coelho (CPTEC, Brazil) Anna Ghelli (ECMWF, UK) Martin Göber (DWD, Germany) Simon Mason (IRI, USA) Marion Mittermaier (Met Office, UK) Pertti Nurmi (FMI, Finland) Joel Stein (Météo-France) Yuejian Zhu (NCEP, USA)

4. 4 FDPs and RDPs Sydney 2000 FDP Beijing 2008 FDP/RDP SNOW-V10 RDP Sochi 2014 MAP D-PHASE Severe Weather FDP Typhoon Landfall FDP

5. 5 Beijing 2008 FDP Real Time Forecast Verification (RTFV) system Fast qualitative and quantitative feedback on forecast system performance in real time –Verification products generated whenever new observations arrive Ability to inter-compare forecast systems 3 levels of complexity –Visual (quick look) –Statistics (quantitative) –Diagnostic (more information)

6. 6 Training In personOnline

7. 7 B08FDP lessons for real time verification Real time verification considered very useful Forecasters preferred scatterplots and quantile-quantile plots Format and standardization of nowcasts products was critical to making a robust verification system Difficult to compare "like" products created with different aims (e.g., QPF for warning vs hydrological applications) Verification system improvements –User-friendly web display –More user options for exploring results

8. 8 SNOW-V10 Verification strategy –User-oriented verification for Olympic period of all forecasts, tuned to decision points of VANOC –Verification of parallel model forecasts for Jan to August 2010 –Nowcast and regional model verification Rich dataset

9. 9 Suggested categories for SNOW-V10 verification

10. 10 Example: Visibility verification

11. 11 Sochi 2014 Standard verification Possible verification innovations: Road weather forecasts Real-time verification Timing of events – onset, duration, cessation Verification in the presence of observation uncertainty Neighborhood verification of high-resolution NWP, including in time-height plane Spatial verification of ensembles User-oriented probability forecast verification

12. 12 Collaboration WWRP working groups THORPEX –GIFS-TIGGE –Subseasonal prediction –Polar prediction CBS –Severe Wx FDPs –Coordination Group on Forecast Verification SRNWP COST 731 ECMWF TAC subgroup on verification measures

13. 13 Spatial Verification Method Intercomparison Project International comparison of many new spatial verification methods Methods applied by researchers to same datasets (precipitation; perturbed cases; idealized cases) Subjective forecast evaluations Workshops: 2007, 2008, 2009 Weather and Forecasting special collection http://www.rap.ucar.edu/projects/icp

14. 14 Spatial Verification Method Intercomparison Project

15. 15 Spatial Verification Method Intercomparison Project Future variables –"Messy" precipitation –Wind –Cloud Future datasets –MAP D-PHASE / COPS –SRNWP / European data –Nowcast dataset(s) Verification test bed

16. 16 Publications Recommendations for verifying deterministic and probabilistic quantitative precipitation forecasts Recommendations for verifying cloud forecasts (this year) Recommendations for verifying tropical cyclone forecasts (next year) January 2008 special issue of Meteorological Applications on forecast verification 2009-2010 special collection of Weather & Forecasting on spatial verification DVD from 2009 Helsinki Verification Tutorial

17. 17 Outreach Verification workshops and tutorials –On-site, travelling EUMETCAL training modules Verification web page Sharing of tools http://www.cawcr.gov.au/projects/verification/

18. 18 International Verification Methods Workshops 4 th Workshop – Helsinki 2009 Tutorial 26 students from 24 countries 3 days Lectures, hands-on (took tools home) Group projects - presented at workshop Workshop ~100 participants Topics: –User-oriented verification –Verification tools & systems –Coping with obs uncertainty –Weather warning verification –Spatial & scale-sensitive methods –Ensembles –Evaluation of seasonal and climate predictions

19. 19 5 th International Verification Methods Workshop Melbourne, December 2011 3-day tutorial + 3-day scientific workshop Additional tutorial foci –Verifying seasonal predictions –Brief intro to operational verification systems Capacity building for FDPs/RDPs, SWFDP, etc.

20. 20 climate change New focus areas for JWGFVR research local point regional global Spatial scale Forecast lead time minuteshoursdaysweeksmonthsyearsdecades NWP nowcasts "Seamless verification" - consistent across space/time scales decadal prediction seasonal prediction sub- seasonal prediction very short range Approaches: deterministic / categorical probabilistic distributional other?

21. 21 New focus areas for JWGFVR research Spatial methods for verifying ensemble predictions Neighborhood, scale-separation, feature-based, deformation average rain rain volume maximum rain rain area

22. 22 New focus areas for JWGFVR research Extreme events

23. 23 New focus areas for JWGFVR research Warnings, including timing Success ratio (1-FAR) Hit rate 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 0

24. 24 Thank you