Splinter Session – ‘Space Weather Metrics, Verification & Validation.’ Thursday 20 th Nov., 16: :00 Splinter session - Space weather metrics, verification & validation

Agenda 16:30 Introduction 16:35 Towards a verification framework for forecasts of different centres, Andy Devos, J. Andries, C. Verbeeck, D. Berghmans (STCE-ROB), Belgium. 16:45 Verification of extreme event forecasts, Institute of Space Physics, Peter Wintoft, IRF, Sweden. 16:50 SWENET Index Quality Statistics & Database Assessment, Alexi Glover, ESA. 16:55 “Performance Verification of Solar Flare Prediction Techniques: Present Status, Caveats, and Expectations”, Manolis Georgoulis, Academy of Athens, Greece. 17:00 The use of modified Taylor diagrams for comparing geophysical models, Matthew Angling, University of Birmingham, UK. 17:10 Translating verification experience from Meteorology to Space Weather, Suzy Bingham, Met Office, UK. 17:20 Lessons learned from CCMC-led community-wide model validation challenges. Outlook on international coordination of M&V activities, Maria Kuznetsova, CCMC, USA. 17:30 Discussion. 18:00 Finish. Splinter session - Space weather metrics, verification & validation

Aims · To understand current techniques used in space weather validation, verification & metrics. · To discuss key physical parameters & products which require validation &/ verification. · To ascertain the validation, verification & metric techniques required to move forward. · To determine the support required by the community. Splinter session - Space weather metrics, verification & validation

Definitions Validation: process of determining the degree to which a product or service (including potentially software & associated data) accurately represent the real world from the perspective of the intended use(s). E.g. the accuracy of the output of a model compared with ‘truth’ data. Verification: process of determining that a system or service (including potentially software, implementation & associated data) perform as expected. E.g. for redeployment of model in a different location—to confirm that a model accepts the full range of inputs, etc. Metrics: statistical parameter. Scientific metrics are related to specific key parameters e.g. Skill scores, Probability Of Detection (POD), False Alarm Ratio (FAR). Application metrics include scientific metrics, extend to overall service performance & are essentially KPIs (Key Performance Indicators), e.g. accuracy & confidence in a service.

Splinter session - Space weather metrics, verification & validation Topics covered in last year’s plenary session on ‘Forecast Verification’ Forecast verification applied by forecast providers; Future development of forecast verification; communicating verification results and requirements between users and providers Metaverification -- evaluating performance measures; Magnetic and other space weather indices; Ground effects of space weather; Ionospheric and magnetospheric processes; Solar and interplanetary data.

Presentations... Splinter session - Space weather metrics, verification & validation

Discussion topics (1) What metrics and validation techniques are required in the current space weather landscape? (2) What are the key challenges currently in model and forecast benchmarking? (3)What direction should the space weather community be taking? (4)What actions can agencies and organisations take in order to support a wider space weather validation effort? (5)How to establish agreed realistic model/service targets to encourage targeted development and prototyping? (6)What targeted actions would encourage groups not currently involved to further participate in space weather validation activities?