1. Veronika Eyring (DLR, Germany) With input from Peter Cox (University of Exeter, USA) Peter Gleckler (PCMDI, USA) Duane Waliser (JPL, USA) Sabrina Wenzel (DLR, Germany) 4 th WGNE workshop on systematic errors in weather and climate models Met Office, Exeter 15-19 April 2013 Process-Oriented Evaluation of Earth System Models

2. ESM Evaluation/ Chart 2 Main Conclusion: … A long way still to go. … Requires a community effort to make it happen.

3. ESM Evaluation/ Chart 3 Overview I.Uncertainties in Earth System Model (ESM) Projections and Examples for Process-oriented Evaluation and Errors  Examples from the Chemistry-Climate Model Validation (CCMVal) Activity  Carbon Cycle (CMIP5) II.Towards a More Quantitative Evaluation: Performance Metrics  Examples of Performance Metric Studies III.Emergent Constraints: Priorities for Model Development and Evaluation?  Seasonal cycle versus long-term climate change timescale  Statistical Frameworks for Model Weighting IV.Related International Efforts  e.g., MIPs, obs4MIPs, WGNE/WCRP metrics panel, diagnostic tools and repository V.Summary, Future Emphases and Needs

4. ESM Evaluation/ Chart 4 I. Uncertainties in Earth System Model (ESM) Projections and Examples for Process-oriented Evaluation and Errors

5. Multi-Model Averages and Uncertainties in Climate Models IPCC, AR4 Global averaged surface air temperature change Models are averaged because the multi-model mean is thought to be better than a single model. Is this always the case? Or can we do this better? Total Column Ozone Anomalies over Antarctica (Oct) WMO, 2011 CO 2 concentration simulated by C 4 MIP models IPCC, AR4

6. Models are increasing in complexity and resolution - From AOGCMs to Earth System Models with biogeochemical cycles - Atmospheric Chemistry https://www2.ucar.edu/news/understanding-climate-change-multimedia-gallery

7. ESM Evaluation/ Chart 7 Lower stratospheric ozone and temperature trends in CMIP5 Eyring et al., JGR, 2013 1979–2005 past 2006-2050 future  Important for example for detection and attribution studies (Santer et al., PNAS, 2013)  Process-oriented evaluation not possible from data collected at ESG, since only O3 was collected.

8. Process-oriented CCM evaluation CCMVal approach to Chemistry-Climate Model (CCM Evaluation) Eyring et al., BAMS, 2005

9. CCMVal-1 (WMO, 2007) Projections of total column ozone (Oct Antarctic) and inorganic chlorine (Cl y ) Eyring et al., 2007 Eyring et al., 2010 CCMVal-2 (WMO, 2011)

10. Anav et al., 2013 CO 2 Flux: Means (1993-2005) and SD In the NH over land almost all models systematically underestimate the sink, reflecting absence of sinks due to nitrogen deposition or poor representation of forest regrowth. In the tropics CMIP5 models do not capture the strong land carbon source, reflecting a coarse parameterization reproducing forest wild fires and deforestation. GLOBAL: Most ESMs reproduce the global ocean and land sink but much larger scatter for global land sink. SINK SOURCE

11. Possible Framework for ESM evaluation

12. ESM Evaluation/ Chart 12 II. Towards a More Quantitative Evaluation: Performance Metrics A performance metric is a statistical measure of agreement between a simulated and observed field (or co-variability between fields) which can be used to assign a quantitative measure of performance (“grade”) to individual models.

13. Grades Performance Metrics: Examples for quantitative evaluation from Chemistry-Climate Model Validation Activity (CCMVal) Waugh & Eyring, ACP, 2008; SPARC CCMVal, 2010 good bad Inorganic Chlorine Cl y Diagnostics Performance Metrics

14. Model evaluation: Carbon cycle performance metrics Anav et al., 2013 PDF based skill scores (mean state and IAV) Surface Temperature and precipitation show general good agreement globally and in the SH and NH, poorer skills in the tropical region. Good scores for global NBP, but low scores in the NH. Very poor skills for Gross Primary Production (GPP) and Leaf Area Index (LAI), both at global scale and in all the sub-domains because these models do not have nutrient limitations and any ozone impact on carbon. NBP

15. Model performance physical climate: revisiting Gleckler et al. (2008) portrait plot with CMIP5 Relative global Root Mean Square Error (RMSE) in climatological annual mean poor good avg See Poster by Peter Gleckler on WGNE/WGCM Metrics Panel

16.  Single model performance index: gauge overall model performance by combining performance metrics that are considered to be important for a particular application. Can be used to define model weights in a multi-model ensemble. Single Model Performance Index Reichler and Kim, BAMS, 2008 Average I 2 of all models within one model group. Multimodel mean taken over one model group. NCEP/NCAR reanalyses.

17. ESM Evaluation/ Chart 17 III. Emergent Constraints: Priorities for Model Development and Evaluation?

18. Emergent Constraint: Seasonal Cycle Physical Climate 1900-2200 April 20th century Apr-May  Large intermodel variations in the strength of snow albedo feedback (SAF) in climate change in the NH in April are nearly perfectly correlated with comparably large intermodel variations in feedback strength in the context of the seasonal cycle. Hall and Qu, GRL, 2006  Feedback strength in the real seasonal cycle can be observed and compared to models.  These mostly fall outside the range of the observed estimate, suggesting many models have an unrealistic snow albedo feedback in the seasonal cycle context.

19. Emergent Constraints: IAV Carbon Cycle Cox et al., 2013 See Poster by Sabrina Wenzel  Future climate change decreases land carbon storage, but large uncertainty in the sensitivity (Friedlingstein et al., 2006).  This leads to uncertainty in future CO 2 levels and how to achieve a mitigation target.

20. ESM Evaluation/ Chart 20 Multiple Diagnostic Ensemble Regression (MDER) applied to CCMVal-1 and -2 simulations Karpechko et al., JAS, subm., 2013 Emergent Constraints: Processes Atmospheric Chemistry More examples discussed at Workshop on Emergent constraints in Exeter, March 2013 -> Talk by Chris Jones on Friday

21. ESM Evaluation/ Chart 21 IV. Related International Efforts

22. CCMValC4MIPILAMBMAREMIPLUCIDAMIPCMIPCFMIPAEROCOMOCMIPAOMIPSIMIPPILPS International Model Intercomparison Projects (MIPs) Many MIPs evaluating individual model components, but a systematic evaluation across all components of an ESM yet missing.

23. Observations for Model Intercomparison Projects (obs4MIPs): - Facilitating the use of Satellite Data to Evaluate Climate Models - Obs4MIPs 1.Use the CMIP5 simulation protocol (Taylor et al. 2012) as guideline for deciding which observations to stage in parallel to model simulations. Target: monthly avg (e.g. OMON, AMON, LMON) products on 1 o x1 o grid 2.Convert Satellite Observations to be formatted exactly the same as CMIP Model output CMOR output, NetCDF files, CF Convention Metadata 3.Includes a 6-8 page Technical Note describing strengths/weaknesses, uncertainties, dos/don’ts regarding interpretations comparisons with models. (at graduate student level) 4.Hosted side by side on the ESG with CMIP5 Jet Propulsion Laboratory California Institute of Technology

24. ESMVal (Earth System Model Validation) High Altitude and LOng Range (HALO) Mission - DLR Project - HIAPER (High-performance Instrumented Airborne Platform for Environmental Research) Pole-to-Pole Observations (HIPPO) of Carbon Cycle and GHG Study NEW: High-altitude and long-range research aircrafts HIAPER HALO SEP 2012 Promoting the Synergy of Models and Observations

25. Beth Ebert (BMRC) – JWGV/WWRP, WMO forecast metrics Veronika Eyring (DLR Germany) – WGCM/SPARC, CCMI, CMIP, ESMs Pierre Friedlingstein (U. Exeter) – IGBP, carbon cycle, ESMs Peter Gleckler (PCMDI), chair – WGNE, atmosphere, ocean Simon Marsland (CSIRO) – WGOMD, ocean Robert Pincus (NOAA) – GEWEX/GCSS, clouds/radiation Karl Taylor (PCMDI) – WGCM, CMIP5, atmosphere Helene Hewitt (U.K. Met Office) – polar ocean and sea-ice WGNE/WGCM Climate Model Metrics Panel An effort to advance the routine evaluation of climate models This panel aims at making results from routine performance metrics more accessible, and in doing so clarify their limitations. It also seeks to gradually facilitate the incorporation a diverse set of more in-depth performance tests. http://www-metrics-panel.llnl.gov/wiki -> See poster by Peter Gleckler

26. The metrics panel package of routine metrics Simple package to offer modeling groups, providing them with the ability to easily benchmark their models against others. This package includes carefully selected observational data, a few very simple codes, and a database of results for all CMIP3 and CMIP5 models. Distribution to modeling groups expected within 3-4 months, with a survey requesting feedback for improvement. Other efforts are underway to develop analysis codes that will be available to the research community (e.g., EMBRACE). A Community-wide diagnostic & performance metrics code repository? At this site scientists involved in climate model evaluation are encouraged to contribute codes that can be used to compute metrics and associated diagnostics. One goal of this collection is to facilitate the sharing of analysis packages (ranging from routine calculations to advanced or novel efforts). It is hoped that this repository might facilitate an increasing openness to climate model evaluation. The WGNE/WGCM metrics panel package and code repository -> See poster by Peter Gleckler

27. FP7 EMBRACE Diagnostic Tool (DLR, SMHI & EMBRACE partners in collaboration with NCAR, PCMDI, GFDL) Open Source: Python Script that calls NCL (NCAR Command Language) Input: CF compliant NetCDF model output (CCMVal, CMIP, etc.) Observations: Can be easily added Extensible: easy to (a) read models (b) process output [diagnostic] with observations and (c) use a standard plot type (e.g. lat-lon map) Gettelman et al., GMD, 2012 Current developments include Essential Climate Variables, e.g.  Sea-Ice  Temperatures  Water Vapor  Radiation  CO2  Ozone Monsoon Diagnostics (from MetOffice) More Observations (e.g., obs4MIPs) Goal: Standard namelists to reproduce certain reports or papers (e.g., Ch8 AR4, Ch9 AR5)

28. Confidence in Models:  Built on physical and biogeochemical principles.  A broad range of climate characteristics is evaluated and the models do well in several important aspects. Some systematic errors remain in ESMs, e.g.  Seasonal cycle (e.g., snow-albedo feedback)  Variability (NH atmosphere-land CO 2 flux)  Representation of specific processes (e.g., transport diagnostics) Advantages of a quantitative multi-model evaluation demonstrated, e.g.  E.g., quantitative assessment of model improvements for single model and generations of models (e.g. CCMVal-1 vrs CCMVal-2, CMIP3 versus CMIP5). Several international activities in place to move ahead ESM evaluation  WGNE/WGCM metrics panel  Model Intercomparison Projects (MIPS)  obs4MIPs  Beginning of sharing of diagnostic tools Summary

29. Evaluation:  Spatial and temporal covariability patterns between Essential Climate Variables (ECVs) across the atmosphere, ocean and terrestrial domains.  An in-depth analysis of the underlying controls of the seasonal cycle, IAV and long-term temporal trends in these variables and processes can help to understand the spread in model projections over the coming decades => priority for model development?  Development of statistical frameworks rather than heuristic model weighting. Model Simulations:  Targeted model simulations and output to improve process-understanding  Coupled and uncoupled runs, better handling of forcings Observations for Model Evaluation:  Fully exploit available observations for model evaluation considering uncertainties.  Identify additional observations to include in obs4MIPs (broader participation, e.g. ESA CCI, CFMIP observations) with guidance from WCRP (e.g. WDAC, WMAC, WGs & projects).  Improve comparability between models and observations (e.g., CCMI insitu expert group, satellite simulators like COSP). Diagnostic and Performance Metrics Tools:  Develop and share common diagnostic tools that routinely run on CMIP output and according observations (obs4MIPs) on the Earth System Grid Federation (ESGF). Future Emphasis and Needs => Benchmark for model evaluation that will over time lead to model improvements