1. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Inter-comparison and Validation Task Team (IV-TT) Report for GOVST Matt Martin, Fabrice Hernandez

2. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Contents Introduction –Objectives of IV-TT –Membership –Main elements of work-plan Santa Cruz workshop outcomes for IV-TT. Recent progress: –Routine class 4 inter-comparison project. –GOV multi-model ensemble project. –Links to GSOP inter-comparison of climate indices. –Other validation work. Summary and future plans.

3. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Introduction Objectives of IV-TT To coordinate and promote the development of scientific validation and inter-comparison of operational oceanography systems. Activities to include: –The definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skill), for physical and biogeochemical parameters. –Setting up specific global and regional inter-comparison experiments. –Define metrics related to specific applications. Liaise with the JCOMM ET-OOFS team for operational implementation. Develop cooperation with CLIVAR/GSOP for climate issues

4. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Introduction Membership Existing members of IV-TT: –Fabrice Hernandez (France) –Matt Martin (UK) –Alistair Sellar (UK) –Hal Ritchie (Canada) –Greg Smith (Canada) –Gary Brassington (Australia) –Jim Cummings (US) –Clemente Tanajura (Brazil) –Peter Oke (Australia) –Pavel Sakov (Norway) –Marina Tonani (Italy) –Avichal Mehra (US) Other members welcome.

5. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Introduction IV-TT work plan 1.Provide a demonstration of inter-comparison and validation in an operational framework: write proposal for production of routine class 4 metrics. implement at participating OOFS. routinely upload data from OOFS to the US GODAE server. generate statistical inter-comparisons on a web-page. once mature, report to ET-OOFS for implementation operationally. continue to test new metrics etc in GOV and regularly update ET- OOFS. 2. Improve inter-comparison and validation methodology for operational oceanography: Obtain feedback from GOVST members participating in dedicated meetings organized by specific communities New metrics outlined during workshops organized by the task team Limited inter-comparison projects between some OOFS, in order to test new metrics.

6. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Santa Cruz workshop IV-TT objectives –GODAE OceanView – CLIVAR-GSOP Workshop on Observing system evaluations and Inter-comparison 13 – 17 June 2011, UCSC, Santa Cruz, CA, USA –Specific goals of the workshop were to: Demonstrate the value of in situ and satellite observations to short- term, seasonal, and decadal forecast systems; Move towards routine monitoring of the global ocean observing system (explore possibilities for “Observation Impact Statements”); Review inter-comparison of class 4 metrics from operational short- range forecast systems and seasonal prediction systems; and Linking inter-comparison and observing system evaluation monitoring activities within GODAE OceanView to those of CLIVAR.

7. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Santa Cruz workshop Inter-comparison and validation presentations Fabrice HernandezGODAE OceanView IV Work Plan Yan Xue*Comparative Analysis of Upper Heat Content Variability from Ensemble Operational Ocean Analyses Alexandre Kurapov*The data assimilation component of the real-time coastal ocean circulation forecast system off Oregon Tong LeeInferring meridional heat transport through meridional volume transport of the AMOC Clemente TanajuraA Sensitivity Study of short-range Forecasts over the Atlantic METAREA V with HYCOM and the Cooper Haines Scheme using different sea surface heights Mario AdaniQuality Assessment of a 1985 - 2007 Mediterranean Sea Reanalysis Stephanie Guinehut Combination of in-situ and satellite observations to monitor the Global Ocean State Gary BrassingtonOceanMAPS version 2 - what did we get from the upgrade? Andrea StortoThe CMCC experience in the context of Global Ocean reanalyses Oscar AlvesAn ensemble ocean data assimilation system for seasonal prediction and its comparison with other state-of-the-art ocean reanalysis Balakrishnan NairReal-time and delayed mode validation for the Indian Ocean Forecasting System (INDOFOS) Gregory SmithEvaluation of CONCEPTS Ice-Ocean Forecasting System Santha AkellaThe GMAO Ocean Retrospective Analysis

8. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Santa Cruz workshop Validation techniques Some commonly used techniques from GOV and coastal groups: –Innovation statistics (bias, RMS, stdev, ACC), for common variables (T, S, SSH, SST) –Time average increments for diagnosing model biases. –Assessing forecast skill using observations and analyses. –SSH variability and position of western boundary currents. Some commonly used techniques from CLIVAR/GSOP groups: –Assessing meridional overturning circulation (AMOC); –Heat transport diagnostics; –Comparisons of the depth of the 20  C isotherm; –Upper ocean heat content for various depth ranges.

9. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Santa Cruz workshop Validation techniques Less commonly used or new assessment techniques (for ocean forecasting systems) include: Assessment of tides in a global system using tide gauges as reference; Altimetry-based analysis of the internal tide (sampling the model in the same way as the data); Comparisons with HF radar data as an independent reference (especially useful when the available velocity data is assimilated); Forecast skill assessments in ensemble forecasting systems, e.g. by comparing spread in the ensemble with the RMSE of the mean; Calculating EOFs of assimilation increments to ascribe various dynamical processes to the model errors; Using Lyapunov exponents to show the scales which are reconstructed by the analysis; Statistical verification of the location of features (e.g. eddies); Verification of Arctic Ocean circulation and investigation of Arctic heat budget; Assessment of volume-integrated heat content in a particular 3D box and calculation of various contributing terms (advective and surface fluxes, data assimilation impact); Validation of geostrophic currents at 1000m depth using the ANDRO data-base (based on Argo trajectories); Comparison of currents with ADCP data from ships and from moored buoys; Comparison of modeled ice thickness with IceSat thickness data..

10. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Santa Cruz workshop Inter-comparison talks Hernandez presented an overview of the IV-TT work plan. Xue presented a detailed comparison of upper ocean heat content (over the top 300m, HC300) from various ocean reanalyses. The questions addressed by the study included: How well is the mean HC300 analysed? What are the impacts of changes in the observing system? How well do the reanalyses capture inter-annual and multi-decadal variability? What climate indices should be monitored? What is the role of HC300 in predictability of ENSO, PDO, NAO, …? Also inter-comparisons carried out by the PEODAS group for assessing their system.

11. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Yan Xue Climate Prediction Center /NCEP Mean Heat Content in 1985- 2009 obs 1985 obs 2008

12. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Santa Cruz workshop IV-TT breakout 1.Discussions of the routine class 4 inter-comparison project => changes to the proposed method to ensure consistency of observations and format. 2.Initial discussion of a proposed project to produce a GOV multi-model ensemble, which we decided to pursue. 3.Discussions of ways in which the GOV groups could get involved in the CLIVAR/GSOP inter-comparison of climate indices. These three initiatives were discussed further yesterday…

13. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 1. Class 4 inter-comparison Overview –Aim is to develop a system for routine inter-comparison of the operational ocean forecasts contributing to GOV. –This will be done in order to facilitate improvements to the systems and to help understand where our systems should focus development. –Comparisons of forecasts against observations was chosen for this so that we don’t have to deal with excessive data volumes. –These data will be made available for inter-comparison: in common formats; for a core sub-set of the various data types which the systems assimilate; to enable assessment of forecasts.

14. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 1. Class 4 inter-comparison Overview –The core data types agreed on for use in the inter-comparison, which are now being implemented are: –SST from in situ drifting buoys, using the US GODAE data-set. –SST from L3 AATSR data-set. –In situ T & S profile data from MyOcean (produced by Coriolis). –Altimeter SLA data from MyOcean (CLS/Aviso). –Model data exchange will be done through the US GODAE server. All files will be encrypted. –The project proposal document has been updated to include a step-by- step guide of how groups can contribute to the inter-comparison. –The document was distributed to the IV-TT and put on the GOV web-site.

15. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 1. Class 4 inter-comparison Overview UKMO download obs files and produce model analysis, forecast, persistence and climatological information at the obs locations in netCDF format. US GODAE server Each OOFS downloads the UKMO files and replaces the model data with those from their system. Any participating group can download files and inter-compare the results. Produce a core set of plots for the GOV web-site (password protected). Observation inputs ~5 days behind real time

16. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 1. Class 4 inter-comparison Current status Production of SST files at UKMO now being done routinely and uploaded onto the US GODAE server – set up by Andy Ryan and Alistair Sellar. Other data types: –code written and tested for T & S profiles, and altimeter SLA data. –Will get T & S profiles, and SLA transferred to US GODAE server soon (profiles in next week or so, SLA before end of 2011). Canadians already started downloading data and plan to start producing the files soon. Mercator, Bluelink, NRL, NCEP and REMO also plan to participate soon. Now setting up web-pages (currently internally) to display information from these class 4 files. These could be used as a basis for an external web-page to display inter- comparison information. If so this would be password protected.

17. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 1. Class 4 inter-comparison Example results from FOAM In Situ SST Profile Temperature RMSE o C

18. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 1. Class 4 inter-comparison Prototype web pages

19. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 2. GOV multi-model ensemble Overview –In the past, for particular incidents (e.g. oil spills), groups have requested sharing of ocean forecasts from GOV partners in order to provide ensemble information. –A proposal was made at the Santa Cruz workshop to begin generation of a GODAE OceanView multi-model ensemble. –There was a general consensus that this should be done and some practical aspects were discussed at the workshop. Yesterday we had further discussions. –A number of the OOFS now have model data available externally and routinely on OpenDAP or ftp servers.

20. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 2. GOV multi-model ensemble Proposal –In order to make the exchange of the data an easier process in the case of emergencies, and to perform some scientific studies on the use of multi-model ensembles, it was agreed that some pilot studies will be carried out: In the Australian region by CSIRO (Peter Oke). Globally by NCEP (Hendrik Tolman). –It was agreed to use analyses and daily mean forecasts out to 6-days of SST, SSS, SSH, surface u and v, and sea-ice concentration and thickness. Also proposed to include mixed layer depth, although difficulties with common definition. –First step is to collect information on access for this data and distribute to the participating groups.

21. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 3. GOV/GSOP climate indices inter- comparison –The production of climate indices from GODAE OceanView systems was discussed at the Santa Cruz workshop. –Mercator are involved in the GSOP inter-comparisons and produce various climate indices from their 1/4  - and 1/12  -resolution models, which has proved useful. –For a number of the GODAE OceanView systems, the priorities are the class 4 and ensemble inter-comparison projects discussed before. –However, it was agreed that we should produce the surface climate indices as part of the multi-model ensemble as a minimum (and interested groups could provide other information if desired). –In addition, it was agreed that some of the systems would produce upper ocean heat content estimates on a monthly basis on a 1 degree grid, and provide them to Yan Xue.

22. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Other validation issues discussed yesterday The XBT community would like to know the impact of their data on the GOV systems. Some groups have agreed to examine particular repeat lines (e.g. near Brazil, near Australia) and to inter-compare model data with the XBTs, e.g.: REMO will compare results to XBT lines from Brazil. CSIRO will compare results to XBT lines near Australia. Tide gauges. Nicolas Ferry presented interesting results from a validation exercise with Mercator. It was agreed that this would be useful to include in the framework of the class 4 inter-comparison project. But on a slower time-scale (once per year). The data-set used so far can be distributed to interested groups. Will need to distribute a list of the location of the tide gauges so that the model information can be calculated routinely.

23. GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011 Summary and future work Workshop at Santa Cruz helped to: Provide overview of validation done at the various centres. Refine plans for class 4 inter-comparison. Propose and discuss GOV multi-model ensemble. Make stronger links with GSOP groups. Class 4 inter-comparison is progressing. General agreement on the GOV multi-model ensemble for surface fields, and a way forward with some pilot studies (globally and around Australia). Agreement from some systems to begin contributing to the inter- comparison of upper ocean heat content organised within GSOP.