1. 1 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute XXII Olympic Winter Games Feb - Mar 2014 Pertti Nurmi WMO WWRP JWGFVR aka Joint Working Group on Forecast Verification Research Acknowledgements : FROST-2014 Expert Team + Evgeny Atlaskin, Sigbritt Näsman pertti.nurmi@fmi.fi Forecast Verification Activities of Sochi 2014 Winter Olympics ( FROST-2014 ~ Forecast and Research in the Olympic Sochi Testbed )

2. 2 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute XXII Olympic Winter Games Feb - Mar 2014 Finnish Meteorological Institute Sochi

3. 3 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute XXII Olympic Winter Games Feb - Mar 2014 Finnish Meteorological Institute

4. 4 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute XXII Olympic Winter Games Feb - Mar 2014 Finnish Meteorological Institute

5. 5 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute XXII Olympic Winter Games Feb - Mar 2014 Finnish Meteorological Institute Observation stations Roshydromet - 6 Avalanche-protection service - 5 Sport venues - 12 Olympic sport venues © Laura RusSki Gorki Rosa Khutor Sanki

6. 6 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute THE issue… on FROST-2014 verification local point regional ( global ) Spatial scale Forecast aggregation time minuteshoursdays NWP nowcasts very short range Deterministic + EPS Wide range of scales, models *, methods, variables, observations … … Difficult to define a general verification setup to cover everything, systematically … … Including end-products to customers  athletes  Impacts … XXII Olympic Winter Games Feb - Mar 2014 Finnish Meteorological Institute * models aka forecast contributors

7. 7 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Issues to consider / take into account … #1 All FROST-2014 contributors verify their own output  General practice ! … BUT can be biased  NOT sufficient / adequate Therefore, also centralized (HMC Russia) AND ”external” verification … to validate verification results Two complementary approaches (like Vancouver Olympics SNOW-V10) a)High temporal resolution Point Verification  User needs : FDP* Users require thresholds for their decisions  Sports activities and events take place at, and along, points Official (final) End-Products as well as objective direct model output a)Research-oriented verification, Detailed model comparisons : RDP* XXII Olympic Winter Games Feb - Mar 2014 * FDP = Forecast Demonstration Project / RDP = Research Demonstration Project (WMO terminology)

8. 8 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Issues to consider / take into account … #2 Details of all available models/systems + observations  Forecast lead times and projections  Enhanced observation network with N (?) stations  Hi-fi site-specific (point) observations highly important On-line forecast production from all contributors  Common data base structure - ftp site - for storage  ”Freeze” the FDP system at certain stage XXII Olympic Winter Games Feb - Mar 2014 Need to “freeze”? When to “freeze”? “ No changes are allowed in the last 3-4 months before the Olympics ” Standardized formats for fc & observation storage ! (Close to) Real-time verification  soon as data(sets) become available Retrospective verification also  Comprehensive post-event verification possible only after all fc/obs archived and QC’d  Two differing results (?) Analyses used as “additional truth” ?  Pitfalls, independence issue  Usefulness – applicability - availability of model independent analyses ( e.g. INCA )

9. 9 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Issues to consider / take into account … #3 Pointwise verification of all forecasts, tuned at all forecast = decision points a)”Baseline quality” from a coarse model as reference b)Various hi-res models from contributors at various (time/space) resolutions c)Statistical adaptations, if there are any (?) d)Official End-Products by the Olympics forecaster team ! e)Inter-comparisons between, and added value, (a) to (d)  IMPACT analysis Define each sports site/event relevant observing station(s) and forecast variables and thresholds  Underway ! Use of verification software packages R, MET, etc … MET, adapted for FROST-2014 (COSMO-RU forecasts) Roshydromet : Internal RDP verification (A. Muravev) FMI : run own “tuned” verification system locally (more on this later)  Input data (fc + obs) via common ftp site XXII Olympic Winter Games Feb - Mar 2014

10. 10 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Issues to consider / take into account … #4 Formal evaluation period : 15 January – 15 March (2 months) High-impact  What is hi-impact ?  Thresholds Sample size issue (statistical inference; confidence intervals) Timing (error) of events – onset, duration, cessation Stratification vs. Aggregation  Need both ! New categorical verification measures : SEDS, SEDI … Spatial verification (CRA, SAL, MODE, FSS …) probably NOT  Applicability/availability of remotely sensed data  Prob. too big efforts ! Case studies  Hard to generalize results  fall under RDP Speciality : FMI Road Weather forecast verification  Next slide  Under externally funded “CoMoSeF” project XXII Olympic Winter Games Feb - Mar 2014

11. 11 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Sochi RWM : First model run 24.8.13

12. 12 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute FROST-2014 : Weather parameter categories & thresholds FROST-2014 “Concept paper” Final definition of events vs. categories Assure availability of deterministic products, contributors Assure availability of EPS, prob. products, contributors Identification of users, decision-makers Identification of FDP vs. RDP verification activities Identification of verification measures / techniques ! →Finalization underway XXII Olympic Winter Games Feb - Mar 2014

13. 13 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute ParticipantsDeterministic modelsEPS VerificationComment HMC, RussiaCOSMO-RU 7 km & 2 km COSMO-RU ~ 1 km COSMO-RU 2 km + official end-products IRAM, Russia+ 4 hr ”nowcasts” ARPA SIMC, Italy COSMO-S-14 7 km Env. CanadaGEM 25 km downscaled to 1 km & 250 m ZAMG, AustriaINCA 1-hourly analysesAladin LAEF NOAA, USANMMB 1 kmNMMB 7 km HIRLAM consortium GLAMEPS Harmon EPS KMA, S-KoreaGlobal 25 km downscaled High resolution 1 km FMI, Finland HARMONIE : 1 km Road Weather Model on selected Sochi roads (CoMoSeF project) Close-to-real-time and retrospective objective point verification Model output at common FROST ftp server in agreed format Official final forecasts by Olympics forecasters Requires online availability of all / most forecasts and local observations FDP: in blue RDP: in green XXII Olympic Winter Games Feb - Mar 2014 FROST-2014: Contributors - Models FMI verification software package + several Nowcasting systems

14. Operational Verification Software @ FMI a 2-minute intro to… Finnish Meteorological Institute

15. Some Properties & Features Web-based; Interactive; Menu- driven; Real-time; 2 languages Daily forecast vs. observation comparison Time-averaged (monthly, seasonal, annual) statistics All basic - and some novelty - verification measures Comparisons : Forecasters’ E-P vs. model; model vs. other models Database coverage : Gridded data + stations over Finland … and Sochi Finnish Meteorological Institute

16. 16 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Result presentation format Zoom Master selection menu Options 1.Producer 2.Parameter 3.Analysis hour 4.Lead time 5.Period type 6.Verif. Measure 7.Presentation 8.Station group User interface: Web browser default page

17. 17 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute http://dev.meteor.fmi.fi/verif/ Idea to adjust to FROST-2014

18. 18 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Master selection menu FROST-2014 Forecasters’ official FCs COSMO - 2.2 km COSMO – 7 km GEM-Downscaled 1 km HARMONIE GLAMEPS HARMONEPS ALADIN-LAEF etc…

19. 19 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Summarizing some issues … #1 Local vs. “external” verification  Contributors “do their own thing”, anyway (Close to) Real-time verification (RTFV)  Visualization of fc-obs time series and inter-comparison on-the-fly  Start aggregating verification statistics already before the beginning of the Games  late 2013 Guarantee observation quality, quantity, homogeneity and availability  Make best use of new observation types  Observation uncertainty  Avoid rejection of good observations (in DA) Event-based verification – Definition of the event - Thresholds  Individual events vs. statistics User-focused (FDP) vs. Model-focused (RDP) verification XXII Olympic Winter Games Feb - Mar 2014

20. 20 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute Summarizing some issues … #2 Wide(r) use of diagnostic verification Use of latest/new scores and measures Verification of high resolution EPS Use of “verification packages” Intuitive, easy-to-understand user-focused verification products and procedures Societal aspects  THE IMPACT Issue Verification training : self-learning tools  No time for face-to-face tutorial ! Improved verification reporting  Joint papers by FROST-2014 group Finalize timeline of actions  Time is short, as always …  Results to be presented at the next ECAM XXII Olympic Winter Games Feb - Mar 2014

21. 21 FROST-2014 Verification activities pertti.nurmi@fmi.fi Finnish Meteorological Institute XXII Olympic Winter Games Feb - Mar 2014 Finnish Meteorological Institute Thank You (Sochi, 2013; Photos © Nikolai Bocharnikov) Other FROST-2014 papers : D. Kiktev : EMS2013-400, Tue 08:45 T. Bazlova : EMS2013-110, poster, Mon-Tue A. Montani : EMS2013-166, Fri 15:15

22. 6 th International Verification Methods Workshop 13-19 March 2013, New Delhi, India Tutorial Session: March 13-15, Scientific Workshop: March 17-19 Highlights recent advances in the theory and practice of verification of weather and climate forecasts and warnings Targets participants from research, operational, and forecast user communities Includes a 3-day WMO verification tutorial session and a 3-day scientific program. Potential topics: Verification of high impact weather forecasts and warnings including heavy rain Verification of ensembles and probability forecasts Spatial forecast verification Climate projection evaluation Intra-seasonal and seasonal forecast verification Tropical cyclone verification Aviation forecast verification Propagation of uncertainty User issues including communicating verification to decision makers Verification tools