Presentation is loading. Please wait.

Presentation is loading. Please wait.

Forecast Verification Research Barbara Brown, NCAR With thanks to Beth Ebert and Laurie Wilson S2S Workshop, 5-7 Feb 2013, Met Office.

Similar presentations

Presentation on theme: "Forecast Verification Research Barbara Brown, NCAR With thanks to Beth Ebert and Laurie Wilson S2S Workshop, 5-7 Feb 2013, Met Office."— Presentation transcript:

1 Forecast Verification Research Barbara Brown, NCAR With thanks to Beth Ebert and Laurie Wilson S2S Workshop, 5-7 Feb 2013, Met Office

2 2 Verification working group members Beth Ebert (BOM, Australia) Laurie Wilson (CMC, Canada) Barb Brown (NCAR, USA) Barbara Casati (Ouranos, Canada) Caio Coelho (CPTEC, Brazil) Anna Ghelli (ECMWF, UK) Martin Göber (DWD, Germany) Simon Mason (IRI, USA) Marion Mittermaier (Met Office, UK) Pertti Nurmi (FMI, Finland) Joel Stein (Météo-France) Yuejian Zhu (NCEP, USA)

3 3 Aims Verification component of WWRP, in collaboration with WGNE, WCRP, CBS (Joint between WWRP and WGNE) Develop and promote new verification methods Training on verification methodologies Ensure forecast verification is relevant to users Encourage sharing of observational data Promote importance of verification as a vital part of experiments Promote collaboration among verification scientists, model developers and forecast providers

4 Relationships / collaboration 4 WGNE SDS-WAS YOTC HyMeX Subseasonal to Seasonal Prediction Polar Prediction WGCM WGSIP TIGGE SWFDP CG-FV SRNWPCOST-731

5 5 FDPs and RDPs Sydney 2000 FDP Beijing 2008 FDP/RDP SNOW-V10 RDP FROST-14 FDP/RDP MAP D-PHASE Severe Weather FDP Typhoon Landfall FDP

6 6 SNOW-V10 Nowcast and regional model verification at obs sites User-oriented verification –Tuned to decision thresholds of VANOC, whole Olympic period Model-oriented verification –Model forecasts verified in parallel, January to August 2010 Status –Significant effort to process and quality-control observations –Multiple observations at some sites observation error

7 7 Wind speed verification (model-oriented) Visibility verification (user-oriented)

8 8 FROST-14 User-focused verification Threshold-based as in SNOW-V10 Timing of events – onset, duration, cessation Real-time verification Road weather forecasts? Model-focused verification Neighborhood verification of high-resolution NWP Spatial verification of ensembles Account for observation uncertainty

9 9 Promotion of best practice Recommended methods for evaluating cloud and related parameters

10 10 Promotion of best practice Verification of tropical cyclone forecasts 1.Introduction 2.Observations and analyses 3.Forecasts 4.Current practice in TC verification – deterministic forecasts 5.Current verification practice – Probabilistic forecasts and ensembles 6.Verification of monthly and seasonal tropical cyclone forecasts 7.Experimental verification methods 8.Comparing forecasts 9.Presentation of verification results

11 Verification of deterministic TC forecasts 11

12 Beyond track and intensity… 12 Precipitation (MODE spatial method) Track error distribution TC genesis Wind speed

13 13 Promotion of best practice Verification of forecasts from mesoscale models (early DRAFT) 1.Purposes of verification 2.Choices to be made a.Surface and/or upper-air verification? b.Point-wise and/or spatial verification? 3.Proposal for 2 nd Spatial Verification Intercomparison Project in collaboration with Short-Range NWP (SRNWP)

14 14 Spatial Verification Method Intercomparison Project International comparison of many new spatial verification methods Phase 1 (precipitation) completed –Methods applied by researchers to same datasets (precipitation; perturbed cases; idealized cases) –Subjective forecast evaluations –Weather and Forecasting special collection Phase 2 in planning stage –Complex terrain –MAP D-PHASE / COPS dataset –Wind and precipitation, timing errors

15 15 Outreach and training Verification workshops and tutorials –On-site, travelling –SWFDP (e.g., east Africa) EUMETCAL training modules Verification web page Sharing of tools

16 16 5 th International Verification Methods Workshop Melbourne 2011 Tutorial 32 students from 23 countries Lectures and exercises (took tools home) Group projects - presented at workshop Workshop ~120 participants Topics: –Ensembles and probabilistic forecasts –Seasonal and climate –Aviation verification –User-oriented verification –Diagnostic methods and tools –Tropical cyclones and high impact weather –Weather warning verification –Uncertainty Special issue of Meteorol. Applications in early 2013

17 Seamless verification 17 Seamless forecasts - consistent across space/time scales single modelling system or blended likely to be probabilistic / ensemble climate change local point regional global Spatial scale forecast aggregation time minuteshoursdaysweeksmonthsyearsdecades NWP nowcasts decadal prediction seasonal prediction sub- seasonal prediction very short range

18 "Seamless verification" – consistent across space/time scales Modelling perspective – is my model doing the right thing? –Process approaches LES-style verification of NWP runs (first few hours) T-AMIP style verification of coupled / climate runs (first few days) Single column model –Statistical approaches Spatial and temporal spectra Spread-skill Marginal distributions (histograms, etc.) Perkins et al., J.Clim. 2007

19 "Seamless verification" – consistent across space/time scales User perspective – can I use this forecast to help me make a better decision? –Neighborhood approaches - spatial and temporal scales with useful skill –Generalized discrimination score (Mason & Weigel, MWR 2009) –consistent treatment of binary, multi-category, continuous, probabilistic forecasts –Calibration - accounting for space-time dependence of bias and accuracy? –Conditional verification based on larger scale regime –Extreme Forecast Index (EFI) approach for extremes JWGFVR activity –Proposal for research in verifying forecasts in weather-climate interface –Assessment component of UK INTEGRATE project

20 Questions What should be the role of JWGFVR in S2S? –Defining protocols? Metrics? –Guidance on methods? –Participation in activities? –Linking forecasting and applications? What should be the interaction with other WMO verification activities? E.g., Standardized Verification System for Long-range Forecasts (SVS-LRF); WGNE/WGCM Climate Metrics Panel How do metrics need to change for S2S? How do we cope with small sample sizes Is a common set of metrics required for S2S? 20

21 Database comments Database should be designed to allow easy access for –Applications –Verification Will need observations for evaluations and applications –Will these (or links to these) be included in the database? –Lack of obs can be a big challenge / detriment to use of the database Access to data –For applications and verification often will not want a whole field or set of fields –Also may want to be able to examine time series of forecasts at points –Data formats and access can limit uses 21

22 Opportunities! New challenges –Methods for evaluating extremes Sorting out some of the thorny problems (small sample sizes, limited observations, etc.) Defining meaningful metrics associated with research questions Making a useful connection between forecast performance and forecast usefulness/value –Application areas (e.g., precipitation onset in Africa) –A new research area Using spatial methods for evaluation of S2S forecast patterns 22

23 23 Thank you

Download ppt "Forecast Verification Research Barbara Brown, NCAR With thanks to Beth Ebert and Laurie Wilson S2S Workshop, 5-7 Feb 2013, Met Office."

Similar presentations

Ads by Google