1. Eidgenössisches Departement des Innern EDI Bundesamt für Meteorologie und Klimatologie MeteoSchweiz The challenge to verify operational weather warnings Tanja Weusthoff and Marco Arpagaus EMS, 14.09.2011 MeteoSwiss official warnings (24h acc. precip.) for 4th September 2011

2. 2 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Introduction 152 warning regions 5 level 8 different hazards  current project: development of an application for an automatic verification of weather warnings

3. 3 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Introduction specifications: develop an automatic verification for operational warnings, which allows a certain tolerance (space and time) use synergies (e.g. with GIN, a common platform for all natural hazards in Switzerland: http://www.gin-info.ch/index.html) motivation: replace current manual (subjective) verification in order to release resources prove the quality of official warnings single official voice (SOV, since January 2011)  distribution via media for level 4 and 5

4. 4 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Challenges how to evaluate the usefulness of warnings without knowing the needs of individual users (and their cost/loss) how to interprete „tolerant results“ and what should be communicated to the users Precautions causes Costs Having no protection results in Losses

5. 5 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Challenges representativity of observations account for feedback of authorities small statistics (rare events) > 2000 m < 800 m e.g. station Magadino / Cadenazzo (203 m asl) is only wind station for two warn regions with complex terrain (308 and 309)

6. 6 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Basic Concept event-oriented verification, warnings as binary events tolerant in space, time and threshold distinguish two types of warnings SHORT: short-term events (e.g. thunderstorms) AKKU: accumulated events (e.g. 24h precipitation accumulation) event definition and evaluation is (in principle) the same within each group; differentiate between basic verification (strict) and detailed verification verification per warning region, summary for specific regions or whole Switzerland

7. 7 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Example: AKKU Rain  level 3-5 Snowfall (lowlands and mountains)  level 3-5 Snowmelt  level 3-5 Heat wave  level 3 snowmelt cannot be verified due to a lack of observations; for snow and rain use of radar data (spatial information) and psychrometer temperature

8. 8 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Event-Definition each warning is an event; duration of warning at least as long as accumulation period duration observation: consider hourly 24h, 48h and 72h sum; important is first threshold exceedance (t2) Hit  An event is observed during a valid warning and the first threshold exceedance (t2) occurs at least accumulationPeriod/3 hours (i.e. 8h,16h,24h) after the beginning of the warn event (ts) and not later then the end of the warn event (te). Miss  A threshold exceedance is observed without an active warning or the first threshold exceedance occurs bevor accumulationPeriod/3 hours after the beginning of the warning (ts). False Alarm  A warning has been issued, but no threshold exceedance has been observed. AKKU t2 - ti > tv? ti issue time WarnEvent (ts to te) tst1 t2te ObsEvent (t1 to t2) acc/3

9. 9 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Evaluation A: Basic-verification Present results as: POD FAR FBI (TS)  derived from contingency table AKKU Event observedNo event Warning issuedA (Hit)B (False Alarm) No warningC (Miss) D (Correct Rejection) Evaluation B: detailed verification in principle like Evaluation A (Hit, Miss, False Alarm) introduce additional category combination of Hit, Miss und False Alarm  see definitions on next slide Missing-D-problem: what is a „non-event“?

10. 10 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Evaluation B: detailed verification (1.) Specifications Miss + Hit + False Alarm  the first threshold exceedance (t2) occurs maximal tshift hours after the end of the warn event (te) or less than accumulatioPeriod/3 (8,16,24h) but more than accumulationPeriod/4 (6,12,18h) after the start of the warning (ts). ti issue time WarnEvent (ts to te) ts t2te ObsEvent (t1 to t2) (2.) AKKU ti issue time WarnEvent (ts to te) tst1 t2te ObsEvent (t1 to t2) t2 - te < tshift acc/4 acc/3

11. 11 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Evaluation B: detailed verification Presentation of results: „perfect“ hit „useful“ combined categories including a hit „bad“ false alarm, miss AKKU „good“... adapted from DWD

12. 12 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Tolerance Allow tolerances in: threshold: LowHit = 90% of threshold time: variation of time components (e.g. tshift) space: if possible, take into account neighbouring warn regions apply tolerances to basic and detailed verification SHORT: possibility to request a minimum leadtime tv for the warning (i.e. a hit is only possible if the warning has been issued at least tv (= t1 - ti) hours before the observed event, otherwise it is classified as a miss) SHORT + AKKU

13. 13 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Application  Flow Diagram obs DataBase  hourly data per warn region warn DataBase  warn events store warn and obs events on a monthly basis store evaluation results for each event on a monthly basis present results per region single event … Java Application AWV derive obs events perform event-based verification for each setting and each region aggregate results and calculate scores for specific period

14. 14 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Observations (surface stations): 24 h acc. MeteoSwiss official warnings (24h acc. precip.) for 4th September 2011 A qualitative example… Level 3 warning: WarnIssue04.09.2011 08:56 WarnStart 04.09.2011 12:00 WarnEnd 05.09.2011 12:00 24h acc. precip. 35 mm/24h 50 mm/24h Bern Luzern precipitation analysis for 4th September 2011 (prel.)

15. 15 Verification of Weather Warnings | EMS 2011 Tanja Weusthoff Thanks for listening …