1. A Regional Climate Model Evaluation System: Facilitating the Use of Contemporary Satellite and Other Observations for Evaluating Regional Climate Model Fidelity D. E. Waliser 1,2, J. Kim 2, C. Mattmann 1,3, C. Goodale 1, A. Hart 1, P. Zimdars 1 and P. Lean 1 National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology Pasadena, California www.nasa.gov Copyright 2010. All rights reserved. Understand uncertainty in projections Figure 1. The role of model evaluation in the model development process and uncertainty estimations. Regional Climate Model Evaluation System (RCMES) : Provide a fast, flexible, comprehensive system to allow easy comparison of climate models with observations. Enable researchers to handle a large volume of data and reduce time taken for model evaluation studies from weeks to hours. Help model developers with cutting-edge observations and diagnostics to evaluate and improve their models. Help end-users understand the uncertainties in climate projections for the regions of interest. Efficient: Fast access to reference data and toolkit User Friendly: Intuitive and transferrable GUI Flexible: Cloud-based architecture Expandable: Easy to add new data/analysis tool Scalable storage solution Regional Climate Model Evaluation System (RCMES) : Provide a fast, flexible, comprehensive system to allow easy comparison of climate models with observations. Enable researchers to handle a large volume of data and reduce time taken for model evaluation studies from weeks to hours. Help model developers with cutting-edge observations and diagnostics to evaluate and improve their models. Help end-users understand the uncertainties in climate projections for the regions of interest. Efficient: Fast access to reference data and toolkit User Friendly: Intuitive and transferrable GUI Flexible: Cloud-based architecture Expandable: Easy to add new data/analysis tool Scalable storage solution Future works: 1.Add additional reference datasets (e.g., other reanalysis, satellite data, in-situ) 2.Examine remote sensing data for evaluating fine-scale (<10km) regional climate data. 3.Additional metrics calculations and visualizations 4.Improve GUI 5.Use the system to evaluate regional/global climate models associated with National Climate Assessment (NCA), NARCCAP, CMIP5 and CORDEX (Africa and Asia). Reference Hart, A.F., C.E. Goodale, C.A. Mattmann, P. Zimdars, D. Crichton, P. Lean, J. Kim, and D.E. Waliser, 2011: A cloud-enabled regional climate evaluation system. SECLOUD’11, May 22, 2011, Waikiki, Honolulu, HI, USA. Future works: 1.Add additional reference datasets (e.g., other reanalysis, satellite data, in-situ) 2.Examine remote sensing data for evaluating fine-scale (<10km) regional climate data. 3.Additional metrics calculations and visualizations 4.Improve GUI 5.Use the system to evaluate regional/global climate models associated with National Climate Assessment (NCA), NARCCAP, CMIP5 and CORDEX (Africa and Asia). Reference Hart, A.F., C.E. Goodale, C.A. Mattmann, P. Zimdars, D. Crichton, P. Lean, J. Kim, and D.E. Waliser, 2011: A cloud-enabled regional climate evaluation system. SECLOUD’11, May 22, 2011, Waikiki, Honolulu, HI, USA. RCMES overview: Large database (MySQL + Apache Hadoop): Multiple reference datasets from: Satellite remote sensing TRMM (1998-2010) AIRS (2002-2010) MODIS Cloudiness Analysis CPC precipitation, CRU precipitation, 2-m air temperatures Assimilation SWR (SNODAS; JPL&U. Colorado) Reanalysis ERA-Interim (e.g. U(p), V(p), q(p), T(p), SLP) Extractors: Process data from various data formats into a common database schema. Library of statistical metrics: Python routines with plug-ins in other languages (Fortran, c, idl) to calculate and plot standard metrics of model performance. (e.g. Bias, RMS error, Anomaly Correlation, Probability Distribution Functions). RCMES overview: Large database (MySQL + Apache Hadoop): Multiple reference datasets from: Satellite remote sensing TRMM (1998-2010) AIRS (2002-2010) MODIS Cloudiness Analysis CPC precipitation, CRU precipitation, 2-m air temperatures Assimilation SWR (SNODAS; JPL&U. Colorado) Reanalysis ERA-Interim (e.g. U(p), V(p), q(p), T(p), SLP) Extractors: Process data from various data formats into a common database schema. Library of statistical metrics: Python routines with plug-ins in other languages (Fortran, c, idl) to calculate and plot standard metrics of model performance. (e.g. Bias, RMS error, Anomaly Correlation, Probability Distribution Functions). 1 Jet Propulsion Laboratory, California Institute of Technology; 2 JIFRESSE, UCLA; 3 University of Southern California Background: Why model evaluation? Climate model projections play a crucial role in developing plans to mitigate and adapt to climate variations and change for sustainable developments. Assessing model performance is an important step in linking climate simulation quality to projection uncertainty and then to climate change impacts assessments. Uncertainties propagate according to model hierarchy Bias correction is based on model evaluation Determination of the weights in multi-model ensemble Model evaluation is also a fundamental part of model development and improvement (Figure 1). Background: Why model evaluation? Climate model projections play a crucial role in developing plans to mitigate and adapt to climate variations and change for sustainable developments. Assessing model performance is an important step in linking climate simulation quality to projection uncertainty and then to climate change impacts assessments. Uncertainties propagate according to model hierarchy Bias correction is based on model evaluation Determination of the weights in multi-model ensemble Model evaluation is also a fundamental part of model development and improvement (Figure 1). RCMES High-level technical architecture RCMED (Regional Climate Model Evaluation Database) A large scalable database to store data in a common format RCMET (Regional Climate Model Evaluation Toolkit) A library of codes for extracting data from RCMED and model and for calculating evaluation metrics Raw Data: Various Formats, Resolutions, Coverage Metadata Data Table Common Format, Native grid, Efficient architecture Common Format, Native grid, Efficient architecture MySQL Extractor TRMM MODIS AIRS SWE ETC Soil moisture Extract OBS data Extract RCM data RCM data user choice Regridder Put the OBS & RCM data on the same grid for comparison Regridder Put the OBS & RCM data on the same grid for comparison Metrics Calculator Calculate comparison metrics Metrics Calculator Calculate comparison metrics Visualizer Plot the metrics Visualizer Plot the metrics URL User’s own codes for ANAL and VIS. Data extractor (Fortran binary) Data extractor (Fortran binary) Sample Graphical User interface Select model data Select Variable 2-m temperature Precipitation OLR (TOA) Cloud fraction 10m wind speed Next > TRMM AIRS level III gridded ERA-Interim URD SNODAS Select Reference Dataset Next > Select data period Next > Select Data Timestep Daily Monthly Seasonal Annual Next > Select Spatial Grid Reference Data Model Next > Select Metrics Mean bias RMSE Pattern correlation PDF Similarity score Coeff. of Efficiency Next > Select Plots Map Time series Process > Evaluation of the Simulated Cold Season Hydrology in California WRF; Oct 2008 – Mar 2009; NCEP Final Analysis forcing Evaluation of the Simulated Cold Season Hydrology in California WRF; Oct 2008 – Mar 2009; NCEP Final Analysis forcing WRF T2 (K): 00UTCAIRS T2 (K): Ascending passes (1:30PM)Bias (K): WRF-AIRS Seasonal-mean 2-m Air Temperature Season-total Precipitation (mm): Multiple Reference Data Bias (mm): WRF-TRMM WRF TRMM CPC Bias (mm): WRF- CPC Issues: Remote sensing data: Satellite fly-over timing Sensor footprints Multiple REF data: Differences between REF datasets Reference data intercomparison Observational uncertainty Issues: Remote sensing data: Satellite fly-over timing Sensor footprints Multiple REF data: Differences between REF datasets Reference data intercomparison Observational uncertainty Evaluation of the CORDEX-Africa Multi-Model Ensemble Preliminary 20-year runs; 1989 –2008 Evaluation of the CORDEX-Africa Multi-Model Ensemble Preliminary 20-year runs; 1989 –2008 ENS Overland mean (mm/day) RMSE (mm/day) Spatial Variability of the Precipitation Climatology using Taylor diagram Precipitation Annual Cycle in 6 Regions using Portrait diagram RMSE Correlation Intuitive presentation schema can facilitate intercomparison of multiple models For more information, please email duane.waliser@jpl.nasa.gov