1. Inter-comparison and Validation Task Team Breakout discussion

2. Discussion topics Overview of work-plan Inter-comparison routine inter-comparison of forecast metrics routine inter-comparison of climate indices multi-system ensemble approach Validation techniques 1 ½ hours overall. ~ 20 minutes each

3. Overview of IV-TT work plan 1. Provide a demonstration of inter-comparison and validation in an operational framework: write proposal for production of routine class 4 metrics. implement at participating OOFS. routinely upload metrics from OOFS to the US GODAE server. after a few months, meet up to discuss progress. once mature, report to ET-OOFS for implementation operationally. continue to test new metrics etc in GOV and regularly update ET- OOFS. 2. Improve inter-comparison and validation methodology for operational oceanography: Obtain feedback from GOVST members participating in dedicated meetings organized by specific communities New metrics outlined during the workshop organized by the task team Limited inter-comparison projects between some OOFS, in order to test new metrics.

4. Class 4 inter-comparison Agreed at GOVST II meeting in 2010 Reference data to use for comparisons: – SST: surface drifters (USGODAE) and L3 AATSR (MyOcean) – SL : tide gauges (Coriolis) and along-track SLA data (CLS/Aviso) – T/S profile : Argo GDAC (USGODAE/Coriolis) Participants: – Bluelink, HYCOM, Mercator, FOAM already agreed. – RTOFS, C-NOOFS/CONCEPTS also expressed interest. – Any others? To set-up and run the generation of the inter-comparison plots: – USGODAE will host data – UK Met Office can produce basic regional RMSE/bias/correlation plots (other participating groups invited to investigate/develop new metrics using the database). Homogenising format of files: – Written in proposal document – Code for generating Class 4 files written at UKMO and can be made available.

5. Class 4 inter-comparison Could produce basic monthly plots of RMSE and ACC vs forecast time in different regions. Discuss how to make use of the class 4 files. Type of averaging for statistics: vertical, horizontal and time. Which regions Routine plots, more detailed assessments, which diagnostics should be produced? How to present results: Taylor diagrams, RMS/ACC vs lead time, ….

6. Climate indices from GOV systems Appropriate to produce these from GOV systems? Which climate indices should be produced from GOV systems? E.g. – MOC -> T. Lee presentation – SST -> Y. Xue presentation – Temperature anomalies at depth -> Y. Xue presentation – 20 deg isotherm depth -> Y. Xue presentation – Heat transport -> T. Lee presentation – Heat content estimates -> Y. Xue presentation – Sea-ice extent -> TOPAZ /CONCEPT validation framework.s Need for reanalysis to put real-time results in context? How to report and who to?

7. Multi-model ensemble Bowler et al., 2008. – Poor man’s ensemble is an effective method for ensemble forecasting, and often better than ensemble prediction systems based on one model. NWP centres do this (for ensemble systems): Thorpex Interactive Grand Global Ensemble (Bougeault et al., 2010) for research purposes. Also done in the GHRSST Multi-Product Ensemble (analysis only): – Run on a daily basis taking in various SST analyses (10 contributing analyses) – Interpolates onto a common grid and calculates ensemble median and standard deviation – Comparison to Argo (independent), shows the median to be better than any of the contributing analyses. – Spread in ensemble may be useful for uncertainty estimation but needs further investigation.

8. Multi-model ensemble Do a similar thing using GOV systems? Which variables? – Could start with SST, SSS, SSH, surface u and v, sea-ice concentration. Which processes do we want to represent and at which length scale? Common grid? Forecast length/frequency? Who would run such a system?

9. Validation techniques New techniques? Which variables? – T, S, SSH, sea-ice variables, u, v, (heat) transports, MOC. – Biogeochemistry? New observation data-sets? – The role of objective analyses of data, EN3, CORA, ARMOR3D High resolution aspects. – Lagrangian metrics. – Upper ocean metrics. What should be adopted more widely?

11. Updates to work-plan Include new validation methodology: Routine class 4 inter-comparison project updates: Inter-comparison of climate indices: Include plan for work on GOV Multi-Model Ensemble?