Steps towards evaluating the cost-benefit of observing systems Rebecca Reid and John Eyre Met Office, UK WMO Impact Studies Workshop Shanghai, China 10-13 May 2016

Outline Motivation Approach Some results Discussion

Acknowledgements Richard Marriott Andrew Lorenc Larry Morgan Mike Thurlow Helen Buttery Malcolm Kitchen Mike Molyneux Steve Stringer Jim Trice Colin Chatters Biz Sykes Jon Turton Sarah North Dave Edwards

Motivation

The role of observations for National Meteorological Services NMS weather and climate services observations people computing “… behind every weather, water and climate condition forecast, every disaster mitigated, and every prediction debated, are the observational data” (WMO RA-V, 15th Session, May 2010, General summary) © Crown copyright 2007

What will WIGOS deliver? Observing systems “Applications” Services sondes NWP Weather Water Climate IR+MW climate monitoring GPS-RO other other strong links and “clean” observational data sets © Crown copyright 2007

Approach

The cost-benefit chain benefit per impact M$ cost M$ benefit impact per cost observing system Application Areas services users direct users of observations indirect users of observations © Crown copyright 2007

Approach Long-term aim: Quantitative evaluation of cost-benefit of observing systems Short-term approach: Measure Impacts of observations in global NWP using Forecast Sensitivity to Observations Impact (FSOI) metric Costs of observing systems operated by the Met Office are known; costs of global observing systems estimated These data used to calculate Impact per Cost for each observing system

Data assimilation 4D-Var FSOI in Met Office global NWP Analysis Observations Forecast “error” T+24 M Data assimilation 4D-Var Forecast T+24 M Forecast T+30 Forecast “error” T+30 Change in forecast error due to each observation !! Change in forecast error (due to obs) Adj.M Change in forecast error due to analysis increments © Crown copyright 2007

FSOI: data for January 2016 total impact, by observation type © Crown copyright 2007

FSOI: data for April-July 2013 total impact, by observation type © Crown copyright 2007

FSOI: data for April-July 2013 impact per observation © Crown copyright 2007

FSOI: data for January 2016 impact per observation © Crown copyright 2007

Impact per cost in global NWP Observation categories – as WMO Vision for the GOS Platforms … and their costs Instruments … and their cost-fractions Data types Cost, for each obs-type Impact, for each obs-type  Impact per cost, for each obs-type For more detail: Eyre J and Reid R. Cost-benefit studies for observing systems. Met Office Forecasting Research Technical Report No.593, August 2014. © Crown copyright 2007

Costs of observing systems for observations used in global NWP Current estimates and guesses Space-based EUMETSAT programmes – estimates - to be improved Other agencies – guesses – based on similar EUMETSAT programmes Surface-based Costs for networks funded or part-funded by UK Numbers of observations: global total, UK-funded total  UK costs scaled up to estimate global costs For more detail: Eyre J and Reid R. Cost-benefit studies for observing systems. Met Office Forecasting Research Technical Report No.593, August 2014. © Crown copyright 2007

Network costs – Met Office, UK Observation type Direct costs (£K) Services received (£K) Total annual cost (£K) Drifting buoy 35 10 45 Moored buoy 600 325 925 Ship 200 330 530 SYNOP station 750 3350 4100 Wind profiler 100 65 165 GPS water vapour 80 115 Radiosonde (land station) 550 500 1050 Observation type (programme) Total annual contribution (£K) Aircraft (E-AMDAR) 285 Ship radiosondes (E-ASAP) 200

Impact per cost (for Met Office global NWP) Met Office costs v. impact of Met Office observations Select the exact observations that we pay for What about programme contributions and “free” data? Met Office costs v. impact of global observations Takes into account benefit from collaboration “Cost to the world” v. impact of global observations Guess “cost to the world” by scaling up known costs 18

Results

impact of Met Office observations Met Office cost v. impact of Met Office observations *Two versions of same graph *Three variables *Note the bar size represents a percentage 20

impact of global observations Met Office cost v. impact of global observations 21

impact of global observations ‘Cost to the world’ v. impact of global observations Metop-A+B AMSU-A Metop-A+B IASI Satellites aircraft Ground-based drifting buoys

Discussion

Limitations of FSOI approach This FSOI metric measures: impact of observations on 24-hour forecast impact of observations in the presence of all other observations Other uses of observations in global NWP: Non-assimilated fields (SST, sea-ice, …) Verification Quality control of other observations Bias correction of other observations Use of observations for other applications Nowcasting Climate monitoring … Shorten these slides

Other considerations These results could guide further, more detailed studies of particular observation types This analysis is presented as a first step towards evaluating “cost:benefit” of observing networks, bearing in mind: limitations of the metric limitations of cost estimates

Extension to other Applications Areas and to services Global NWP has appropriate metrics – FSOI, OSE Metrics needed for other Application Areas (Aas): High-resolution NWP possible, but more challenging Other AAs (Nowcasting, climate monitoring, …) … likely to be less objective, but don’t worry – better than nothing! From AAs to services Challenging Link to work quantifying benefits of NMSs © Crown copyright 2007

Conclusions - next steps Improve estimates/guesses of observing system costs, Generalise the results to more than one NWP centre and more than one period Promote development of appropriate metrics for other Application Areas Extend impact-per-cost assessments to other parts of the cost-benefit chain, … and eventually to an integrated assessment of cost-benefit over many applications and services © Crown copyright 2007

Thank you! Any questions?

Observing networks – marine Observation type Met Office stations Drifting buoys The Met Office is responsible for ~25 annually (there are around 1250 worldwide) Moored buoys Network of 11 stations in UK waters Ships Approximately 250 manual VOS and 40 AWS equipped ships were supported by the Met Office and operational during the period of study Moored buoys Drifting buoys Ships

Observing networks – land Observation type Met Office stations SYNOP stations Approximately 290 land stations in UK network Wind profilers 7 stations in UK network GPS water vapour Approximately 120 stations in UK network Radiosondes (land stations) 6 stations in UK network (excluding defence stations providing irregular soundings) SYNOP Wind profilers GPS water vapour Radiosondes

Observing networks – programmes E-AMDAR: Aircraft observations E-ASAP: Ships providing radiosonde ascents