1. ESMValTool for Benchmarking Models with ESA CCI Data Mattia Righi 1, Veronika Eyring 1, Axel Lauer 1, Alexander Löw 2, Benjamin Müller 2, Daniel Senftleben 1, Sabrina Wenzel 1, Martin Evaldsson 3, Ulrika Willén 3, and Yoko Tsushima 4 CMUG WP 5.1 Contribution 1 Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Oberpfaffenhofen, Germany 2 Department of Geography, University of Munich (LMU), Germany 3 Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden 4 Met Office, Exeter, United Kingdom CCI CMUG Integration Meeting Munich, 14-16 March 2016

2. Motivation and Goal Model benchmarking initiatives have become increasingly important to evaluate the quality of coupled Earth System Models (ESMs) and to support the model development process. However, ESA data is currently not used in the context of routine model evaluation. Model benchmarking is also important for the Climate Research Groups (CRGs) within the CCI and various CMUG activities. CMUG will therefore establish a standardized model benchmarking approach for the CCI, and will contribute to the development of a community-wide Earth System Model Evaluation Tool (ESMValTool) that is currently developed by different partners in different projects. The goal is to work towards a standardized community based benchmarking toolkit that includes ESA CCI data and that could be used operationally for CMIP6 analysis in the forthcoming years. It is expected that the preparation of the individual ESA CCI datasets in CMIP compliant format for Obs4MIPs as well as the corresponding technical note for Obs4MIPs are created and submitted by the individual ESA CCI teams.

3. Overview ESA CMUG WP 5.1 Tasks The aim of this WP is to contribute to the development of a standardized community based evaluation and benchmarking tool that makes full use of the novel ESA CCI data products and that could be used for CMIP model analysis in the forthcoming years by the climate research community. Within the ESA CCI programme, the ESMValTool will be provided to the Climate Research Groups for their work within the ESA-CCI programme. This task will also provide comparison to other observations that are routinely used in model evaluation, in particular those that are contributed to obs4mips. D5.1 v1: Initial version of the ESMValTool with one ESA CCI dataset for test purposes shared among CMUG partners [DLR, LMU, June 2015]. D5.1 v2: Advanced version of the ESMValTool with ESA CCI datasets and user guide released to CMUG and ESA CCI teams [DLR, LMU, June 2016]. D5.1 v3: Final version of the ESMValTool with 10 ESA CCI datasets and user guide released to wider community [DLR, LMU, June 2017].

4. ESMValTool Version 1.0 released with ESA CMUG contributions http://www.esmvaltool.org/ Eyring et al., Geosci. Model Dev. Discuss, 8, 7541-7661, 2015

5. Current Status: Contributing Institutions (currently ~60 developers from 22 institutions and ~40 users) 1.Deutsches Zentrum für Luft- und Raumfahrt (DLR), Institut für Physik der Atmosphäre, Germany 2.Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden 3.Agenzia nazionale per le nuove tecnologie, l’energia e lo sviluppo economico sostenibile (ENEA), Italy 4.British Atmospheric Data Centre (BADC), UK 5.Centre for Australian Weather and Climate Research (CAWCR), Bureau of Meteorology, Australia 6.Deutsches Klimarechenzentrum (DKRZ), Germany 7.ETH Zurich, Switzerland 8.Finnish Meteorological Institute, Finland 9.Geophysical Fluid Dynamics Laboratory (GFDL) NOAA, USA 10.Institut Pierre Simon Laplace, France 11.Ludwig Maximilian University of Munich, Germany 12.Max-Planck-Institute for Meteorology, Germany 13.Met Office Hadley Centre, UK 14.Météo France, Toulouse, France 15.Nansen Environmental and Remote Sensing Center, Norway 16.National Center for Atmospheric Research (NCAR), USA 17.New Mexico Tech, USA 18.Royal Netherlands Meteorological Institute (KNMI), The Netherlands 19.University of East Anglia (UEA), UK 20.University of Exeter, Exeter, UK 21.University of Reading, UK 22.University of Wagingen, The Netherlands

6. Maintenance, Documentation and Technical Development 1. Maintenance -A subversion controlled repository has been made available to allow the development by multiple users. In addition, a Mantis bug tracking system and a wiki page is provided to facilitate communication and documentation of the tool. -First release of ESMValTool version 1.0 as open source software in December 2015 -The ESMValTool core development team (DLR, LMU, and SMHI) is responsible for maintaining a stable version of the tool in the trunk including quality control and testing. 2. Documentation -ESMValTool documentation paper published in GMDD in August 2015, revisions pending -A first version of the user guide has been published in GMDD and will be further revised for the final version to support the ESA CCI teams in the use of the tool -In addition, ESMValTool developers’ internal wiki page that documents new developments and progress 3. Technical development -First ESMValTool technical workshop has been hold (March 2015, Munich) for the definition of road maps; several additionl meetings between DLR and LMU -Automated testing of diagnostics (work in progress) -Visualize the results through a website (work in progress) -ESMValTool performance improvements (work in progress)

8. Development and integration of ESA CCI data into the ESMValTool ESA CCISatellite Instrument(s) the ESA CCI dataset is based on ESA CCI Data ProductsStatus in ESMValTool Cloud properties (A)ATSR-MODIS-AVHRR (1982-2014) AATSR-MERIS (2002-2012) cc (clh, clm, cll), ctp, cth, ctt, ref, cot, lwp, iwp, cot_ctp_hist2d Under work OzoneGOME, SCIAMACHY and GOME-2 Envisat limb instr. (GOMOS, MIPAS, SCIAMACHY) and ESA Third Party instr. (OSIRIS, SMR, and ACE-FTS) Total column ozone Vertical profiles Prototype AerosolsATSR2 (1996-2003) AATSR (2002-2012) od550aer Other data products Operational Not yet included GHG (CO2)SCIAMACHY/ENVISAT (2002-2012), TANSO/GOSAT (2009-ongoing); XCO2Under work Sea iceSSM/I and AMSR-E (1992-2011) Envisat RA-2 Sea ice concentrations (SIC) Sea ice thickness (SIT) Under work Not yet included Sea level Radar Altimetry (1993-2014) Gridded sea level anomalies Not yet included Sea surface temp. AVHRR, ATSR (monthly 1992-2008) SSTUnder work Ocean colour MERIS, MODIS, SeaWIFS, CZCS, VIIRS, OCM-2, S3-OLCI Chlorophyll-aNot yet included Land cover MERIS, SPOT Vegetation Major cover type fractions Land surface conditions Prototype Fire SPOT Vegetation Burned areaNot yet included Soil moisture (A)ATSR, MERIS, SPOT Vegetation Volumetric surface soil moisture Prototype

9. Performance Metrics calculated with ESA CCI Data [DLR] Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology (1980–2005) computed from the historical CMIP5 experiments. Similar to Figure 9.8 of IPCC AR5 (Flato et al., 2013 and Gleckler et al. (2008)). A similar figure will be produced for selected ESA CCI ECVs using ESA CCI as the reference data set and if available an alternate observational data set for comparison. NEW: ESA CCI total column ozone ESA CCI AOD 550 nm already implemented in ESMValTool version 1.0 Implementation of other ESA CCI datasets planned Clouds & Radiation Aerosol+ XCO2 SIC and SIT Ocean colour SSTs Land Cover Fire Soil Moisture Eyring et al., ESMValTool (v1.0), GMDD, 2015

10. Goal: ESMValTool as one of the CMIP documentation functions to routinely assess the performance of CMIP DECK and the CMIP6 Historical Simulations, running alongside the ESGF Goal: Run ESMValTool alongside the ESGF for Routine Evaluation of CMIP6 Models Eyring et al., CMIP6 Overview, GMDD, 2015 More routine usage of ESA CCI data for ESM evaluation studies through a community based evaluation and benchmarking tool

11. ATMOSPHERE: Clouds [SMHI] (1) CCI Cloud Cover has stronger minima and maxima and larger values at high latitudes over sea compare to the models and CLARA-A2 -Preliminary results using Cloud-CCI and CLARA-A2 cloud cover for the “Performance metrics”. -Annual climatology for observations, reanalysis and historical CMIP5 experiments (1982-2005). -BIAS figs: models compared to CCI-cloud. Relative errors: Cloud CCI upper/CLARA lower triangle -Mean Median Can ERAINT GFDL IPSL MPI LR MPI MR NCEP2 -model model ESM2 ESM2G CM5lr ESMlr ESMmr - ERAINT-CCI -CanESM-CCI- IPSL-CCI- CanESM-CCI - GFDL-CCI- MPImr-CCI - NCEP-CCI -cci -clara

12. ATMOSPHERE: Clouds [SMHI] (2) CCI cloud cover has larger variability than the models... Reference point Cloud_CCI ERA-Interim Preliminary results using ESMvaltool Taylor diagram diagnostics showing the multi-year annual average performance of individual CMIP5 models and the multi-model mean in reproducing ESA- CCI Cloud cover (AVHRRv1.4), for common model and observational period 1982-2005 CLARA-A2

13. ATMOSPHERE: Teleconnections Clouds & SST [SMHI] Preliminary results using ESMvaltool Teleconnection diagnostics for the correlation between Nino3.4 CCI-SST and CCI -Cloud cover, and SST/ Clouds in two AMIP simulations,1982-2008 ANN-mean Cloud-Nino3.4 SST teleconnections CCI SST&Clouds EC-Earth SST&Clouds IPSL SST&Clouds EC-Earth - CCI correlations IPSL - CCI correlations Observed and modelled teleconnection pattern similar, but the models show hint of “double” ITCZ Difference in models and observed correlations

14. ATMOSPHERE: Ozone [DLR] Comparison of total column ozone in CMIP5 models with ESACCI and NIWA -Seasonal climatology and time series for the period 1997-2005 -Overall good agreement between ESACCI and NIWA data

15. Overall good agreement between ESACCI and AERONET Overestimate of AOD in some stations (Sahara, Arabia, South America). Good agreement in marine stations. ATMOSPHERE: Aerosols [DLR] (1)

16. OCEAN GLOBAL Evaluation of aerosol optical depth (AOD) at 550 nm in the CMIP5 models Model spread is large (~0.05 - 0.20). Differerence between MODIS Collection 6 data and ESA CCI product also significant. ATMOSPHERE: Aerosols [DLR] (2)

17. ATMOSPHERE: Greenhouse Gases [DLR] (CO 2 ) Preliminary comparison of column averaged atmospheric CO 2 (XCO 2 ) in CMIP5 models with ESA CCI data: -Data available for 2003-2013 -CMIP5 models tend to overestimate XCO 2 from ESACCI XCO 2 -Averaging kernels not yet used; more detailed comparison to follow. region 60N – 90N

18. Evolution of Arctic (top) and Antarctic (bottom) summer sea ice extent, representing the accumulated sea ice area of all grid cells with at least 15% sea-ice coverage Generally large spread between different sea-ice observational datasets ESACCI data in terms of sea ice extent rather on the high end ESACCI-SSMI and -AMSR relatively similar OCEAN: Sea Ice [DLR]

19. TERRESTRIAL: Soil Moisture [LMU] (1) Comparison of soil moisture spatial patterns and PDFs between models and ESA- CCI soil mosture product (Loew et al., 2013) MODEL BIAS Percentile spatial correlation

20. TERRESTRIAL: Soil Moisture [LMU] (2) Covariability between soil moisture and precipitation dynamics in models and ESA CCI soil moisture product Anomaly correlation between precipitation and soil moisture for satellite CDRs and reanalysis data (Loew et al., 2013)

21. Challenge: no consistent PFT‘s across different CMIP models  comparison of broad classes (tree, shrubs, bare, water); preprocessing using LC CCI user tools( Brovkin et al., 2013) TERRESTRIAL: Land Cover [LMU] (1)

22. Snow has major impact on surface albedo and surface net radiation budget. Major discrepancies in climate models existing (snow albedo, seasosnality) (e.g. Hagemann et al., 2013; Loew et al., 2016) ESA CCI land cover conditions is used to evaluated snow dynamics Surface net radiation difference (model – observations) TERRESTRIAL: Land Cover [LMU] (2)

23. ATMOSPHERE: Clouds Auto-Assess [MetOffice] (1) -Aims: To produce a set of metrics & diagnostics to inform model development -Current status: -Aim to produce a first (alpha) external release to Unified Model partners in April 2016 -Currently assumes native Unified Model file format (pp) -A beta release around the end of the year. In the beta release we hope to be using CF standard names where they exist -Auto-assess Clouds & Radiation to be linked to ESMValTool -Data Format -Function to read standard names have been added -Confirmed to read in CERES-EBAF, ISCCP D2 in netcdf. Aim to read in other data in Obs4MIPs -What should we do with diagnostics with no CF standard names? -Interface -Integration with shallow link to be done for D5.1 v3 June 2017

24. ATMOSPHERE: Clouds Auto-Assess [MetOffice] (2) - Auto-assess Clouds and Radiation summary plot -

25. -ESMValTool documentation (user guide) published -Further important technical improvements identified and work underway (e.g. performance improvements) -Work on the implementation of ESA CCI ECVs so far mostly focused on studying these individually -Next step: inclusion of additional ECVs in performance metrics plot and more in depth analysis -D5.1 v2: Advanced version of the ESMValTool with ESA CCI datasets and user guide released to CMUG and ESA CCI teams [DLR, LMU, June 2016]. -Planned contribution from WP 5.1 to Remote Sensing of Environment Special Issue on Earth Observation of Essential Climate Variables -Performance Metrics Plot with CMIP5 models using ESA CCI datasets as reference dataset and alternate observations where possible -Individual sections for each implemented ESA CCI with details on the analysis and discussion of possible issues for climate model evaluation Summary and Outlook