1. Components, Coupling and Concurrency in the Earth System Modeling Framework N. Collins/NCAR, C. DeLuca/NCAR, V. Balaji/GFDL, G. Theurich/SGI, A. da Silva/GSFC, M. Suarez/GSFC What is ESMF? ESMF provides tools for turning model codes into components with standard interfaces and standard drivers, for use in application ensembles and hierarchies ESMF provides data structures and common utilities that components use for routine services such as data communications, regridding, time management and message logging ESMF Community NSF National Center for Atmospheric Research NOAA Geophysical Fluid Dynamics Laboratory NOAA National Centers for Environmental Prediction NASA Goddard Global Modeling and Assimilation Office NASA Goddard Institute for Space Studies NASA Jet Propulsion Laboratory NASA Goddard Land Information Systems project DOD Naval Research Laboratory DOD Air Force Weather Agency DOD Army Engineer Research and Development Center DOE Los Alamos National Laboratory DOE Argonne National Laboratory University of Michigan Princeton University Massachusetts Institute of Technology UCLA Center for Ocean-Land-Atmosphere Studies Programme for Integrated Earth System Modeling (PRISM) Common Component Architecture (CCA) ESMF Impacts ESMF impacts a very broad set of research and operational areas that require high performance, multi-component modeling and data assimilation systems, including: Climate prediction Weather forecasting Seasonal prediction Basic Earth and planetary system research at various time and spatial scales Emergency response Ecosystem modeling Battlespace simulation and integrated Earth/space forecasting Space weather (through coordination with related space weather frameworks) Other HPC domains, through migration of non-domain specific capabilities from ESMF – facilitated by ESMF interoperability with generic frameworks such as CCA Open Source Development Open source license (GPL) Open source environment (SourceForge) Open repositories: web-browsable CVS repositories accessible from the ESMF website http://www.esmf.ucar.edu http://www.esmf.ucar.edu for source code for contributions (currently porting contributions and performance testing) Open development priorities and schedule: priorities set by users (previously during meetings and telecons, transition to new Change Review Board), web-browsable task lists Open communication: frequent telecons and mailing list discussions Open testing: all tests are bundled with the ESMF distribution and can be run by users Open port status: results of nightly tests on many platforms are web-browsable Infrastructure Timeline: ESMF Path ESMF common modeling infrastructure (~5 year) Technical foundation that allows for organized and exchangeable codes Set of modeling codes that are based on ESMF (~5 year) Community pool of interoperable science components with which to assemble applications Multi-agency and international organization (~5 year) Organizational foundation on which to establish collaborations and set priorities Standards for model and data description (~10 year) Prerequisite for an advanced modeling and collaboration environment that includes knowledge management ESMF part of an established end-to-end community-based modeling environment (~20 year) An Earth System Modeling Environment (ESME) that combines models, data, experiments, collaborative tools and information resources in a way that fosters knowledge sharing and accelerates scientific workflow Computational Characteristics of Weather and Climate Mix of global transforms and local communications Load balancing for diurnal cycle, event (e.g. storm) tracking Applications typically require 10s of GFLOPS, 100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs Required Unix/Linux platforms span laptop to Earth Simulator Multi-component applications: component hierarchies, ensembles, and exchanges Data and grid transformations between components Applications may be MPMD/SPMD, concurrent/sequential, combinations Parallelization via MPI, OpenMP, shmem, combinations Large applications (typically 100,000+ lines of source code) ESMF Current Challenges Process and techniques for software requirements analysis, design and interface review with a VERY large distributed community (installation of commercial software (DOORS) for requirements archival and tracking, ties to testing and tasking) Transition to multi-agency organizational structure Design strategy for advanced grids and regridding; science and math hires to implement new grids and regridding methods Continued improvement of development and test processes Clear, complete, carefully edited documentation and training program materials New collaborations for technical and domain extension assim sea ice ocean land atm physicsdycore assim_atm atmland Seasonal Forecast coupler Hierarchies and Ensembles ESMF encourages applications to be assembled hierarchically Coupling interfaces are standard at each layer Components can be used in different contexts assim_atm Ensemble Forecast assim_atm ESMF supports ensembles with multiple instances of components running sequentially (and soon, concurrently) Platforms Application Example: GEOS-5 AGCM Each box is an ESMF component Every component has standard methods and interfaces to facilitate exchanges New components can be added to the application systematically Data in and out of components are state types that contain fields, bundles of fields Coupling tools include regridding and redistribution methods ESMF Interoperability Experiments Coupling existing models for the first time 3 experiments done CCSM CAM with NCEP analysis GFDL FMS B-grid atmosphere with MITgcm ocean CCSM CAM with MITgcm ocean 5 additional experiments underway, with GFDL MOM4 ocean, LANL CICE, NCAR/NCEP WRF, NASA GMAO ocean and atmosphere 3 NCAR Community Atmospheric Model (CAM) coupled to MITgcm ocean Atmosphere, ocean, and coupler are set up as ESMF components Uses ESMF regridding tools 1 GFDL B-grid atmosphere coupled to MITgcm ocean Atmosphere, ocean, and coupler are set up as ESMF components Uses ESMF regridding tools 2 Temperature SSI import Temperature SSI export Temperature difference Registration Components must provide a single externally visible entry point which will register the other entry points with the Framework. Components can: - Register one or more Initialization, Run, Finalize, and other standard entry points. - Register a private data block which can contain all data associated with this instantiation of the Component; particularly useful when running ensembles. Higher level Comp cmp_register() cmp_run() cmp_final() cmp_init() ESMF Framework Services Public subroutine Private subroutine Key Concepts Standard Methods: Initialize, Run, Finalize Coupling Strategies Hub and spokesPairwise Serial Execution Concurrent Execution Conclusions Component approach powerful for integration of very large applications Allows concurrent development in different domains and at different institutions Allows common bodies of code to be used in different contexts with no change Collective and ongoing design, review, and testing Encourages concentration on science and not computational issues ESMF Component Overhead Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall Run on NCAR IBM Runs done by JPL staff, confirmed by NCEP developers