1. Translating verification experience from meteorology to space weather Suzy Bingham ESWW splinter session, Thurs. 20 th Nov.

2. Current space weather validation, verification & metrics Validation: plans to work with CCMC for TEC validation. Verification: plan to use experience in terrestrial methods; to verify models & forecaster warnings. Application metrics/KPIs: have extended terrestrial infrastructure to monitor space weather systems. Plan to verify for stakeholders. Met Office forecaster Webpages: Enlil & REFM.

3. Verification of forecasts Why verify: to understand quality/accuracy, evidence for stakeholders, forecaster feedback, to further improve, to compare models/methods. Space weather forecasters produce guidance: twice daily. These forecasts include probability forecasts for geomagnetic storms, X-ray flares, high energy protons & high energy electrons. Example probability forecasts

4. Verification of forecasts Metrics/performance indicators used in terrestrial weather forecasting verification for stakeholders (with examples): 1.Severe weather warning accuracy (score: impact level, area covered & validity time) 2.Forecast accuracy (e.g. daily minimum temp accuracy should be +/-2°C) 3.Public value (“How useful are forecasts these days”?) 4.Public reach (“Have you seen any weather warnings in the last few days?”) 5.Service quality (Timeliness scores for model delivery.) 6.Emergency responder value (How satisfied are you with the service?”) 7.Responder reach (Availability of Hazard Manager application to emergency responder community) 8.National capability (95% of lightning location data messages should be available within a certain time.) 9.Milestone achievement (Develop a national electronic weather data archive)

5. © Crown copyright Met Office Verification of warnings time Low Event threshold HIT FALSE ALARM MISS NON- EVENT Warning periodEnd time + late hit period Issue time EARLY HIT LATE HIT Event threshold EARLY LOW HIT LATE LOW HIT LOW HIT Introducing the warnings verification system

6. Gale warning verification St Judes Storm 27-28 October 2013...... a hit!

7. Performance: ROC plot Performance of Forecaster issued Heavy Rainfall Alerts in 2012: © Crown copyright Met Office

8. Application metrics: timely, reliable, robust Timeliness plot for Global Model. Server monitoring to verify system is robust/reliable.

9. Data latency SOHO: EIT 4 channels, LASCO – C2, C3 LASCO C2 & C3 – 12 min cad / ~1-2 hrs latency (but data gaps – sometimes up to 5 hrs). SDO: AIA 12 channels, HMI-magnetogram STEREO A&B – COR1, COR2, EUVI cadence: COR1 – 1hr, COR2 – 15 min, EUVI – 10 min latency – 1-3 hrs (but currently very big data gaps/no data). ENLIL: run every 2 hours Run completed / graphics produced around 4 hrs after model analysis time (ie T=0). WMO Coronograph image requirements, OSCAR webpage

10. Met Office Business Performance Measures (BPMs) The Met Office BPMs are: Forecast accuracy, Growth, Reach, Customer & Service Delivery, Efficiency & Sustainability Excellence. Verification underpins the Forecast Accuracy BPM which is set by government. This BPM is to improve: 1.Global NWP Index 2.UK NWP Index 3.Public Forecasts 4.Customer Forecasts Global NWP Index is compiled from: Mean sea-level pressure, 500 hPa height, 850 hPa wind, 250 hPa wind. Plot showing increase in Global NWP Index which shows an improvement in Global model accuracy.

11. Summary Met Office is currently applying some validation, verification & metrics to space weather models/service (e.g. verifying reliability of service). Met Office is planning to take part in CCMC TEC challenge with initial longitudinal section case studies over Europe. Met Office is planning to adapt terrestrial verification methods to space weather to allow forecasters to understand warning accuracy (e.g. flexible verification system). Met Office would like to provide simple metrics to stakeholders.

12. Useful links WMO space weather observation requirements, Observing Systems Capability Analysis & Review Tool: http://www.wmo- sat.info/oscar/applicationareas/view/25http://www.wmo- sat.info/oscar/applicationareas/view/25 Forecast verification, Centre for Australian Weather & Climate Research: www.cawcr.gov.au/projects/verification/ www.cawcr.gov.au/projects/verification/ http://www.eumetcal.org/resources/ukmeteocal/temp/msgcal/www/english/co urses/msgcrs/crsindex.htmhttp://www.eumetcal.org/resources/ukmeteocal/temp/msgcal/www/english/co urses/msgcrs/crsindex.htm NOAA verification metrics glossary: http://www.nws.noaa.gov/oh/rfcdev/docs/Glossary_Verification_Metrics.pdf http://www.nws.noaa.gov/oh/rfcdev/docs/Glossary_Verification_Metrics.pdf

13. Questions and answers

14. Verification of forecasts Metrics/performance indicators used in terrestrial weather forecasting verification for stakeholders: 1.Severe weather warning accuracy 2.Forecast accuracy 3.Public value 4.Public reach 5.Service quality 6.Emergency responder value 7.Responder reach 8.National capability 9.Milestone achievement 0-2Very PoorWarning was missed or gave very poor guidance to customer, perhaps being classed as a “False Alarm” 3-4Poor Guidance Although a warning was issued it gave poor guidance to the customer 5-7Good Guidance A warning was issued which gave generally good guidance to the customer 8-9Excellent Guidance The warning issued gave excellent guidance to the customer Scoring for quality of sever weather warning.

15. Performance: reliability Performance of Forecaster issued Heavy Rainfall Alerts in 2012: © Crown copyright Met Office

16. Met Office Business Performance Measures (BPMs) The Met Office BPMs are: Forecast accuracy, Growth, Reach, Customer & Service Delivery, Efficiency & Sustainability Excellence. Verification underpins the Forecast Accuracy BPM which is set by government. This BPM for FY13/14 was to improve: 1.Global NWP Index - 101.88 with a stretch of 102.43. 2.UK NWP Index - 123.05 with a stretch of 123.9. 3.Public Forecasts- 12/17 forecast targets met with a stretch of 17/17. 4.Customer Forecasts- 2/3 forecast targets met with a stretch of 3/3. Global NWP Index is compiled from: Mean sea-level pressure, 500 hPa height, 850 hPa wind, 250 hPa wind. Plot showing increase in Global NWP Index which shows an improvement in Global model accuracy.