1. © Crown copyright Met Office Cost benefit studies for observing systems Stuart Goldstraw, Met Office, CBS-RA3-TECO-RECO, 13 th September 2014

2. © Crown copyright Met Office Contents This presentation covers the following areas Background The cost – benefit chain Introducing the FSO Tool Some results for the Met Office Discussion and Next Steps

3. © Crown copyright Met Office Background CBS-14 established IPET-OSDE as part of the structure of OPAG-IOS. WIGOS Network Design Principles, as discussed at the CBS-Ext Session include the following: 5. DESIGNING COST-EFFECTIVE NETWORKS Observing networks should be designed to make the most cost-effective use of available resources. This will include the use of composite observing networks. This presentation is based on the paper presented by John Eyre to IPET-OSDE-1 in March 2014 and the subsequent work undertaken John Eyre and Rebecca Reid.

4. © Crown copyright Met Office Why study cost benefit for observing systems? Society benefits from the Services we provide too them These Services are developed using the outputs obtained from 12 recognised WMO Application Areas Observing Systems are developed and operated to meet the requirements of these Application Areas Observing Systems have costs associated with them The requirements of Society, our Services and Application Areas have evolved and will continue to evolve with time Therefore we need to optimise the investment in observing systems to ensure we meet the changing needs of Society.

5. © Crown copyright 2007 The cost-benefit chain Observing Systems Impact per cost Services Application Areas Society Indirect users of observations M$ cost M$ societal benefit Benefit per impact Direct users of observations Direct users of Services Direct users of Application outputs Remote users of observations

6. 12 WMO Application Areas Global NWP Regional NWP Nowcasting and VSR Forecasting Sub-seasonal and longer range forecasting Aeronautical Meteorology Atmospheric Chemistry Ocean Applications Agricultural Meteorology Hydrology GCOS Climate Applications Space Weather © Crown copyright Met Office

7. The four stages to cost benefit impact The costs of Observing Systems The impact of observations on Application Areas The impact of Application Areas on Services The benefit of Services to Society

8. © Crown copyright Met Office Focus on the first two components The costs of Observing Systems The impact of observations on Application Areas The impact of Application Areas on Services The benefit of Services to Society

9. © Crown copyright Met Office Simplify the challenge further The costs of UK Observing Systems The impact of observations on Global NWP The impact of Application Areas on Services The benefit of Services to Society

10. © Crown copyright Met Office Considering the Global NWP Application Area Developments in Data Assimilation methodologies have allowed the community to develop new observations impact tools. The Forecast Sensitivity to Observations (FSO) tool can provide an indication of the relative impact of observations on forecast skill We also understand the costs of the observing systems we operate Therefore we can determine a relative impact per cost of the observing system, the first two steps in the benefit chain.

11. © Crown copyright Met Office The Adjoint-based FSO Method The change in forecast error at T+24 is entirely due to the assimilation of observations at T+0. Slide courtesy of Richard Marriott

12. © Crown copyright Met Office FSO evaluation – total impact Total impact of observations by type, period April- July 2013 Negative impacts show an improvement in forecast skill

13. © Crown copyright Met Office © Crown copyright 2007 FSO Evaluation – impact per observation Impact per observation by type, period April- July 2013 Negative impacts show an improvement in forecast skill

14. © Crown copyright Met Office Costs of Met Office Observing Programmes in 2013 in US$ AMDAR* ASAP* Drifting Buoys Moored Buoys Ship Obs Surface Synoptic Wind Profilers^ GPS IWV Radiosonde^ $0.47M $0.33M $0.07M $1.53M $0.87M $6.77M $0.27M $0.19M $1.75M

15. © Crown copyright Met Office Impact of global observations and use of Met Office costs

16. © Crown copyright Met Office Firstly considering the total impact Radiosonde, AMDAR and Synops are showing largest impact

17. © Crown copyright Met Office Secondly considering the impact per observation Moored buoys, Ship Obs and Synops showing similar impact

18. © Crown copyright Met Office Finally considering impact per cost AMDAR and Drifting Buoys show high impact per $

19. Distribution of buoy observations © Crown copyright Met Office

20. Distribution of synop and ship obs © Crown copyright Met Office

21. Interpretation of results Impacts must be taken in the context of the total current ‘system’: the observing system; the Data Assimilation system and the Model version. Impacts should be interpreted as "the effective impact which those observations are having in the presence of all other observations" (RM 2013) Impacts only represent the change in error of our chosen forecast metric – i.e. the global moist energy norm on 24- hour forecasts. Impacts should be averaged over many cases, in this case 4 runs a day for 3 months.

22. © Crown copyright Met Office Considering the three layers of detail together provides a fuller evaluation of the value of networks Important to avoid misinterpretation of results: e.g. ‘high impact per cost’ does not imply ‘invest here’ Results may help to advise evolutionary changes in network design, if they are considered in full context. They can provide a ‘top level’ analysis, and be considered alongside individual characteristics of the observation types of interest (e.g. data quality, location, etc) This analysis is presented as a first, tentative step at evaluating ‘cost benefit’ of observing networks – remembering the specificity of the metric Discussion Points

23. © Crown copyright Met Office Improved estimates of the global, or regional costs of component observing systems are required. Results need to be generalised across multiple Global NWP centres. Development of methodologies to determine impacts across the other 11 WMO Application Areas. Development of methodologies to distribute costs of observing systems that serve more than one application area in a weighted manner. Further development of the end to end cost to benefits methodology – building on existing work, some of which has been highlighted this week. What are the next steps

24. © Crown copyright Met Office Acknowledgement This presentation could not have been made with the ongoing work being undertaken by John Eyre and Rebecca Reid and the previous work and advice of Richard Marriot.

25. © Crown copyright Met Office Thank you for your time