1. Earth System Model Evaluation Tool (ESMValTool)
Axel Lauer1, Yoko Tsushima2, Mattia Righi1, Veronika Eyring1, Alexander Löw3, Ulrika Willén4, and Mark Ringer2
1 Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Oberpfaffenhofen, Germany
2 Met Office, Exeter, United Kingdom
3 Department of Geography, University of Munich (LMU), Germany
4 Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden
27 May 2015

2. Motivation – Development of an Earth System Model Evaluation Tool
Facilitate the evaluation of complex Earth System Models, e.g.
quick look at standard diagnostic plots & output diagnostic variables
easy comparison of new simulations (e.g. sensitivity runs or runs with new model versions) with existing runs and with observations
Raise the standard for model evaluation
include additional diagnostics of ongoing evaluation activities so that we do not have to start from scratch each time
implement more observations, account for observational uncertainties
Quick assessment of where we stand with a new set of model simulations via standard namelists that reproduce specific papers, reports, etc.
Traceability and reproducibility
Easy expendability
synergies with ongoing projects to expand the tool (e.g. NCAR CVDP)
useful for model groups & those analyzing models
useful for model development

3. Technical Improvements Schematic Overview of the ESMValTool Structure
Introduction to the ClimValDiaglTool 3

4. Development of an Earth System Model Evaluation Tool
Within EMBRACE: DLR, SMHI & EMBRACE partners in collaboration with NCAR, PCMDI, GFDL
Open Source: Python Script that calls NCL (NCAR Command Language) and other languages (e.g. R, Python)
Input: CF compliant netCDF model output (CMIP standards)
Observations: can be easily added
Extensible: easy to (a) read models (b) process output [diagnostic] with observations and (c) use standard plot types (e.g. lat-lon map)
Current developments include for example
Essential Climate Variables, e.g.
Sea-Ice
Temperatures & water vapor
Radiation
CO2
Ozone
Tropical variability (incl. monsoon, ENSO, MJO)
Southern ocean
Continental dry biases and soil-hydrology-climate interactions (e.g. standardized precipitation index)
Atmospheric CO2 and NO2 budget
Aerosol, clouds
Stratospheric ozone
Goal: Standard namelists to reproduce certain reports or papers (e.g., IPCC AR5 Chapter 9, Massonnet et al., 2012; Anav et al., 2012; Wenzel et al. 2014; Eyring et al., 2013)
Release as open-source software planned for later this year.

5. Running along-side the ESGF
ESMValTool: A community diagnostic and performance metrics tool for routine benchmarking and process evaluation of Earth System Models in CMIP
Similar to Figure 9.7 of AR5
Monsoon Precipitation Intensity and Domain
CMIP5 MMM
Running along-side the ESGF
CMIP5 MMM - OBS
Similar to Figure 9.7 of AR5
Link to projections
Similar to Figure 9.5 of AR5
Similar to Figure 9.24 of AR5
Similar to Figure 9.24 of AR5
Introduction to the ClimValDiaglTool 5

6. Considering Observational Uncertainty in the ESMValTool
Observational uncertainty needs to be considered in the assessment of model performance and should be an integral part of the evaluation tools
The ESMValTool is quite flexible and can consider observational uncertainty in many ways, e.g.:
using more than one observational dataset to evaluate the models
showing the differences between the observational datasets in addition to the difference between the model and the reference dataset
easily usage of an observed uncertainty ensemble spanning the observed uncertainty space, consistent with statistics
Example: HadCRUT4 surface temperatures, 5 (out of 100) realizations

7. WCRP Grand Challenges: (1) Clouds, circulation and climate sensitivity, (2) Changes in cryosphere, (3) Climate extremes, (4) Regional climate information, (5) Regional sea-level rise, and (6) Water availability, plus an additional theme on “biospheric forcings and feedbacks”
Goal
ESMValTool as one of the CMIP documentation functions to routinely assess the performance of CMIP DECK and CMIP6 simulations running alongside the ESGF
Meehl et al., EOS, 2014

8. Work as part of ESA CCI CMUG
Model benchmarking initiatives have become increasingly important to evaluate the quality of coupled Earth System Models (ESMs) and to support the model development process.
However, ESA data is currently not used in the context of routine model evaluation.
Model benchmarking is also important for the Climate Research Groups (CRGs) within the CCI and various CMUG activities.
CMUG will therefore establish a standardized model benchmarking approach for the CCI in close collaboration with the CMIP and WGNE/WGCM Climate Model Metrics Panel, and will contribute to the development of a community-wide Earth System Model Evaluation Tool (ESMValTool) that is currently developed by different partners in different projects – coordinated by DLR.
The goal is to work towards a standardized community based benchmarking toolkit that includes ESA CCI data and that could be used operationally for CMIP6 analysis in the forthcoming years.
It is expected that the preparation of the individual ESA CCI datasets in CMIP compliant format for obs4MIPs as well as the corresponding technical notes for obs4MIPs are created and submitted by the individual ESA CCI teams.

9. WP 5.1 – Subtasks
Contribute to the community-wide development of an Earth System Model Evaluation Tool (ESMValTool) by developing code and integrating diagnostics and performance metrics into the tool
Ensure integration of ESA CCI datasets as well as other ESA datasets (in particular GlobAlbedo or other Glob-products) in the ESMValTool for routine model evaluation and benchmarking.
Provide comparison to other observations that are routinely used in model evaluation, in particular those that are contributed to obs4MIPs (
Provide recommendations of standardized benchmarking to the CMIP and WGNE/WGCM climate model metrics panels.
Visualize the results through a website to enhance visibility of the ESA CCI data. The ESMValTool will be extended with capabilities to visualize the resulting plots on a website.
Support the CCI Climate Research Groups in the use of the ESMValTool by providing a user guide with the goal that the CCI CRGs can use the tool in their own evaluation work.

10. Goal: Performance Metrics calculated with ESA CCI Data
Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology (1980–2005) computed from the historical CMIP5 experiments. Figure 9.8 of IPCC AR5 (Flato et al., 2013).
A similar figure will be produced for selected ESA CCI ECVs using ESA CCI as the reference data set and if available an alternate observational data set for comparison.

11. Baseline Proposal and Batch Options 1 and 2
Atmospheric ECVs (aerosols) [DLR]
Terrestrial ECVs (soil moisture, fire, land cover) [MPI-M]
Ocean ECVs (SST, sea-ice) [DLR, MPI-M]
Batch Options 1 – O5.1
Atmosphere: Ozone, GHG (CO2) [DLR], Clouds [Met Office]
Ocean: ocean colour [MPI-M]
Atmosphere, land and ocean [SMHI]
Batch Options 1 – O5.2
Evaluation of a selected set of CMIP5 [DLR, MPI, Met Office] and CCI4MIP simulations [MPI-M]
Using the results from the ESMValTool, contribute to a report that includes an assessment of:
how well state-of-the-art Earth system models simulate climatological mean, variability and trends of selected ECVs,
how the new observational time series of the ESA CCI change the model evaluation and benchmarking assessment of the models in comparison to observations provided by obs4MIPs,
whether observational uncertainty is still a limiting factor in benchmarking models.
Batch Options 2 – O5.1
Atmosphere: GHG (CH4) [DLR]

12. DLR contributions What has been done so far
1. Maintenance and documentation (Eyring/Lauer/Righi) (WP5.1)
A subversion controlled repository has been made available to allow the development by multiple users. In addition, a Mantis bug tracking system and a wiki page is provided to facilitate communication and documentation of the tool.
A user guide is in preparation that will support the ESA CCI teams in the use of the tool
Work on ESMValTool documentation paper started
Technical improvements are on-going
First release planned for fall 2015
2. Development and integration of ESA CCI data into the ESMValTool (WP3.6, WP3.7, WP5.1, O3.2, O5.1, O5.2)
Atmospheric ECVs: aerosols, Righi/Lauer), ozone and GHG/CO2 (Wenzel), clouds (Lauer/Righi)
Ocean: sea ice (Senftleben)
Work with new ESA’s new 17 year ATSR climate data record (AOD) started
Will be continued in year 2, performance metrics plots will be provided for all CMIP5 models, AOD will be added (also WP3.7)
Cloud products (CLT, LWP) from non-ESA sources (also WP3.7)
O3, CO2, sea ice (also for WP3.6): work with ESA CCI data will be started in year 2

13. Technical ESMValTool workshop 3-4 March 2015, Munich, Germany
Present status and future perspectives of the ESMValTool
Specific questions addressed
What technical improvements can be made that would facilitate broad community development and usage?
How to simplify ESMValTool integration with existing data repositories and the ESGF infrastructure?
What are necessary steps towards a roadmap for ESMValTool enhancement for CMIP6 and beyond?
How to prepare the software release and what is an appropriate license?
→ Technical improvements of the ESMValTool are supported also by ESA CMUG

14. Technical ESMValTool workshop 3-4 March 2015, Munich, Germany
Timeline for submission of first ESMValTool documentation paper
The workshop was of particular importance for the technical development of diagnostics of the ESA CCI datasets and their integration into the ESMValTool, and for technical requirements to run the tool on new CMIP data alongside the ESGF.
Defining the further mid-term roadmap for ESMValTool development on the international level.
A first release of the ESMValTool as open-source software is planned this year as a deliverable of EMBRACE. This release will form an important basis for the work on the ESMValTool as part of ESA CMUG and will allow to include the ESA CCI data in an efficient manner. From ESA CMUG, we are planning to include some selected diagnostics using the Aerosol CCI 17 year ATSR climate data record already in this first release.
→ Technical improvements of the ESMValTool are supported also by ESA CMUG

15. Assessment of CMIP5 models with new ESA’s 17 year ATSR climate data record
As in Fig of IPCC Chapter 9.
AOD integrated over the oceans for CMIP5 models, MODIS and ESA-CCI

16. Clouds
20-yr average LWP ( ) CMIP5 in comparison with UWisc satellite climatology

17. LMU contributions (Alexander Löw)
Technical work
LMU has implemented a flexible python interface into ESMValTool that allows for integration of new diagnostics.
LMU has developed a prototype for flexible automated testing of diagnostics within ESMValTool. This will maximize the quality of diagnostics across all ECVs.
LMU has started to implement a soil moisture diagnostic to integrate ESA CCI soil moisture data (Loew et al., 2013) – to be finalized for first CMUG release of ESMValTool.
Administration
LMU has so far not signed a sub-contract; draft of sub-contract was submitted by Met Office and is currently with LMU administration. Until signature and hiring of staff only minor progress can be expected.

18. SMHI contributions (Ulrika Willén)
First tests using ESA-CCI SST with the ESMValTool
Example plot

19. Metrics of Clouds and Radiation
Yoko Tsushima
ESA CMUG integration meeting May 2015
© Crown copyright Met Office

20. Auto-assess:
Aims: Primary aim is to produce a comprehensive set of metrics & diagnostics to inform model development.
Current status:
Scientific development currently frozen whilst code is re-written in Python using iris library.
Aim to produce a first external release to UM user community in 2015.
Currently assumes native UM file format (pp).
© Crown copyright Met Office

21. Current & planned assessment areas
Global tropospheric circulation
Conservation
Clouds & radiation
Hydrological cycle
Asian monsoon systems
MJO
ENSO
Processes over Africa
Tropical cyclones
Mid-latitude storm tracks and blocking
Stratosphere
Land surface
Aerosols
Ocean
Sea Ice
© Crown copyright Met Office

22. Auto-assess metric plot
© Crown copyright Met Office

23. Cloud Regime Error Metric for present day (CREMpd)
(Williams and Webb 2009, Tsushima et al.,2013)
Tsushima et al 2013
RMSE metric based on cloud clustering of daily ISCCP simulator data
The error can be broken down into different cloud regimes, contributions of frequency of occurrence component and in-regime cloud radiative effects
Code provides as one of CFMIP diagnostics
© Crown copyright Met Office

24. Improvements are seen in mainly in the radiative properties of optically thicker low and high topped cloud regimes rather than frequency of occurrence.
Shallow Mid-level Thin Low level Convective Deep Stratocu
Cumulus Topped Cirrus Transition Anvil Conv
in in-cloud Net CRE error in regime Net CRE RMSE
CFMIP1 -> CMIP5 Change
Change between CFMIP-1 and CMIP5 of in-cloud Net Cloud Radiative Effect (Net CRE) Error in Net CRE RMSE within daily ISCCP simulator cloud regimes in the Tropics
Tsushima et al 2013
© Crown copyright Met Office

25. Cloud Feedback Model Intercomparison Project (CFMIP) CFMIP diagnostics code repository
Klein et al., 2012
Tsushima et al.,2013, Williams and Webb, 2009
Cloud Regime Error Metric for present day (CREMpd)
Qu et al., 2012
Nam et al., 2012
Sherwood et al., 2014
CFMIP encourages climate modelling and diagnostic groups to both use the existing metrics and provide additional relevant codes .

26. Summary and Outlook
An ESM Evaluation Tool is available that facilitates the complex evaluation of ESMs and their simulations (e.g. submitted to international Model Intercomparison Projects such as CMIP, C4MIP, CCMI)
The tool is developed under a subversion controlled repository
Allows multiple developers from different institutions to join the development team
Broad community-wide and international participation in the development is envisaged
Collaboration with the WGNE/WGCM metrics panel
Current extensions
Atmospheric dynamics, biogeochemical processes, cryosphere and ocean
Need for a wide range of observational data to be used with the tool
Observations should ideally be provided with uncertainty and a technical documentation (e.g. similar to those from obs4MIPs) and in CF compliant netCDF
Improving statistical comparison and visualization
Regular releases to the wider international community
Further develop and share the diagnostic tool and routinely run it on CMIP DECK output and according observations (obs4MIPs) alongside the Earth System Grid Federation (ESGF)
Release of ESMVal tool to the public → will contribute to metrics panel code repository
Work towards a full community-tool developed by multiple institutions / countries

27. Thank you!