1. Observing System Evaluation Task Team Report Peter Oke and Gilles Larnicol CSIRO and CLS

2. Talk Outline  2013 GOV review – implications for the OSEval-TT  OSEval-TT membership  Main achievements  Focus areas for annual TT activities/projects  Main priorities  Expected outcomes  Expected investment and return for investment  Annual projects  Synergies between GOV task teams

3. 2013 GOV review  Review held in Nov 2013  OSEval–TT main achievements presented participation to the discussion  Findings from the panel  work to be of high quality  work closely with GOOS/OOPC and Clivar/GSOP  responsiveness were particularly noteworthy

4. 2013 GOV review Recommendations : 1.further articulation of the OSEval work plan within the context of the overall GOV strategy with:  more explicit connection to partners;  greater emphasis on system-wide evaluation. 1.to develop this TT as an objective and authoritative source of advice and evidence on the relevance of the observing system taking account of model and assimilation changes and improvements  link with GOOS/OOPC, JCOMM;  link with Observational bodies as CEOS, GHRSST, OSTST, ARGO, …

5. OSEval-TT membership Core Members:  Magdalena Balmaseda (ECMWF)  Anthony Weaver (CERFACS)  Laurent Bertino (NERSC)  Pavel Sakov (NERSC)  Gary Brassington (BoM)  Jim Cummings (NRL)  Pat Hogan (NRL)  Yosuke Fujii (JMA/MRI)  Villy Kourafalou (Univ. Miami)  Daniel Lea (UKMet)  Matthew Martin (UKMet)  Avichal Mehra (NOAA)  Elisabeth Remy (Mercator-Océan)  Greg Smith (EC)  Clementa Tanajura (Remo)  Zhang Tianyu (NMEFC) Associate members:  Mike Bell (UKMet)  Eric Dombrowsky (Mercator-Océan)  Fabrice Hernandez (Mercator-Océan)  Eric Lindstrom (NASA)  Andreas Schiller (CSIRO)  Hans Bonekamp (EUMETSAT) Co-Chairs:  Peter Oke (CSIRO)  Gilles Larnicol (CLS)

6. OSEval-TT membership Core Members:  Magdalena Balmaseda (ECMWF)  Laurent Bertino (NERSC)  Jim Cummings (NRL)  Yosuke Fujii (JMA/MRI)  Villy Kourafalou (Univ. Miami)  Daniel Lea (UKMet)  Elisabeth Remy (Mercator-Océan)  Greg Smith (EC)  Clementa Tanajura (Remo)  Zhang Tianyu (NMEFC) Associate members:  Andreas Schiller (CSIRO)  Hans Bonekamp (EUMETSAT)  Paul DiGiacomo Co-Chairs:  Peter Oke (CSIRO)  Gilles Larnicol (CLS)

7. Achievements: three community papers

8. GOV OSEval workshop December 10-11th 2014 (Toulouse) : Jointly organised with GSOP-CLIVAR group & E-AIMS FP7 European Project More than 25 abstracts submitted Should be focused on the results analysis and work plan refinement- To be decided now Achievements: OSEval-TT next workshop

9. Priorities  Community engagement - both within GOV, and with the observational agencies. Community activities for us means applying multiple systems to equivalent experiments - and is a way of establishing system-independent, robust results. Workshops e.g., OSEval-GSOP Quarterly newsletter e.g., GOV newsletter, OSEval newsletter, submission to Mercator newsletter Special sessions e.g., AGU, EGU Official OSEval-TT presentation Participation at Obs meetings e.g., Argo Steering team, XBT steering team, OSTST, …

10. Outcomes  Guidance for observational agencies to optimise the GOOS, and convince funders of the value of ocean observations to ocean forecasting and those beneficiaries of ocean forecasting DefenceSearch & rescue Environmental monitoring Oil spill response Oil and gas planning, assessment & approval Fisheries management Ship routing Leisure (yatch racing, non- commercial fishing, idiots trying to paddle small craft between continents)

11. Investment and benefit  It’s hard to motivate people to work for free …  The efficacy of our community activities would be substantially increased if we had a dedicated effort … dedicated PostDoctoral researcher Technically coordinate community activities Analyse community results Disseminate results Cost: $100K- AU$180K Publish in peer-reviewed literature

12. Linkages with other TTs OSEval IV DA Annual project 1 Annual project 2 Annual project 3 Annual project 4 Annual project 1 Annual project 2 Annual project 3 Annual project 4 Annual project 1 Annual project 2 Annual project 3 Annual project 4

13. Linkages with other TTs OSEval IV DA Annual project 2 Annual project 3 OSEval-IV-DA Annual project 1 Annual project 2 Annual project 3 OSEval-IV Annual project 4 Annual project 2 Annual project 3 Annual project 4

14. Annual projects  NRT OSEs   Routine monitoring Routine demonstration of observation impact by disseminating information on what observations are assimilated into GOV systems. This is simply regularly (e.g., quarterly) reporting how many observations of each type (e.g., altimetry, Argo, XBT, TAO, AVHRR SST etc) are assimilated by each forecast system. Information to be provided by each participating forecast systems for every assimilation cycle. Forecast centers are encouraged to routinely compute affordable obs-impact metrics – and analyse results for contribution to the Official OSEval-TT presentation AltimeterArgoXBTAVHRR…… Bluelink3x10 5 300 (2x10 4 )35 (4x10 3 )8x10 5 FOAM3.5x10 5 200 (2x10 4 )50 (5x10 3 )9x10 5 Mercator3.2x10 5 250 (2x10 4 )25 (3.5x10 3 )7x10 5 GIOPS4x10 5 300 (2x10 4 )10 (1x10 3 )7.5x10 5 : Sample only

15. Annual projects  Routine monitoring  Annual community OSEs - COSEs  3 OSEs per year  6 month period  Daily-mean “best-estimate” fields produced each week (i.e., 26 fields provided) along with class 4 fields for the following set of OSEs:  ALL: OSE that assimilates “all” observations;  NONE: OSE that assimilates no observations; and  COSE: OSE that with-holds an agreed set of observations.  Analysed by a central agency

16. Annual projects  Routine monitoring  Annual community OSEs - COSEs ORAS4 FOAM REMO Mercator Bluelink

17. Annual projects  Routine monitoring  Annual community OSEs - COSEs Fujii et al. (2014) – TPOS community paper

18. Annual projects  Routine monitoring  Annual community OSEs - COSEs Initial timeEnd timeData with-heldDue date 7/201312/2013Degraded Argo array (-20%)4/2015 7/201412/2014 X altimeter(s) of the constellation (TBD) 4/2016 7/201512/2015Tropical moorings4/2017 7/201612/2016X% of Argo (TBD)4/2018 7/201712/2017 X altimeter(s) of the constellation (TBD) 4/2019

19. Annual projects  Routine monitoring  Annual community OSEs  Inter-comparison of single-obs impacts; Inter-comparison of single observation impacts: using an array of “idealised background innovations” (representing an difference between a set of hypothetical observations and each system’s background field) participating agencies will compute increments derived from their forecast system. Obs X  variable Y by Z% …

20. Annual projects  Routine monitoring  Annual community OSEs  Inter-comparison of single-obs impacts; System 1 (EnKF) System 1 (EnOI) Sandery et al. (2014) Covariance shown – not increments

21. Summary Annual Projects 1.Routine monitoring 2.Annual community OSEs 3.Inter-comparison of single-obs impacts Ongoing Initiatives  Official OSEval-TT presentation;  Encourage forecast centers to routinely compute affordable obs-impact metrics (e.g., DFS, SRF);  Compare conclusions drawn from OSEs and affordable obs-impact metrics; and  Community OSSE guidance.

22. Strategic Plan document update