Extension of the ESMF for Space Weather Cecelia DeLuca SWW April 7, NSIPP Seasonal Forecast NCAR/LANL CCSM NCEP Forecast GFDL FMS Suite MITgcm NASA GSFC GSI

Outline Overview Application Integration Projects ESMF Status Extension to Space Weather

ESMF Goals 1.Increase scientific productivity by making model components much easier to build, combine, and exchange, and by enabling modelers to take full advantage of high-end computers. 2.Promote new scientific opportunities and services through community building and increased interoperability of codes (impacts in collaboration, code validation and tuning, teaching, migration from research to operations).

What is ESMF? 1.ESMF provides tools for turning model codes into components with standard interfaces and standard drivers 2.ESMF provides data structures and common utilities that components use for routine services such as data communications, regridding, time management and message logging

Each box is an ESMF component Every component has standard methods and interfaces to facilitate exchanges New components can be added to the application systematically Data in and out of components are state types that contain fields, bundles of fields Coupling tools include regridding and redistribution methods Application Example: GEOS-5 AGCM

ESMF Community ◦NSF National Center for Atmospheric Research ◦NOAA Geophysical Fluid Dynamics Laboratory ◦NOAA National Centers for Environmental Prediction ◦NASA Goddard Global Modeling and Assimilation Office ◦NASA Goddard Institute for Space Studies ◦NASA Jet Propulsion Laboratory ◦NASA Goddard Land Information Systems project ◦DOD Naval Research Laboratory ◦DOD Air Force Weather Agency ◦DOD Army Engineer Research and Development Center ◦DOE Los Alamos National Laboratory ◦DOE Argonne National Laboratory ◦University of Michigan ◦Princeton University ◦Massachusetts Institute of Technology ◦UCLA ◦Center for Ocean-Land-Atmosphere Studies ◦Programme for Integrated Earth System Modeling (PRISM) ◦Common Component Architecture (CCA)

ESMF Impacts ESMF impacts a very broad set of research and operational areas that require high performance, multi-component modeling and data assimilation systems, including: Climate prediction Weather forecasting Seasonal prediction Basic Earth and planetary system research at various time and spatial scales Emergency response Ecosystem modeling Battlespace simulation and integrated Earth/space forecasting Space weather (through coordination with related space weather frameworks) Other HPC domains, through migration of non-domain specific capabilities from ESMF – facilitated by ESMF interoperability with generic frameworks such as CCA

Infrastructure Timeline: ESMF Path ESMF common modeling infrastructure (~5 year) Technical foundation that allows for organized and exchangeable codes Set of modeling codes that are based on ESMF (~5 year) Community pool of interoperable science components with which to assemble applications Multi-agency and international organization (~5 year) Organizational foundation on which to establish collaborations and set priorities Standards for model and data description (~10 year) Prerequisite for an advanced modeling and collaboration environment that includes knowledge management ESMF part of an established end-to-end community-based modeling environment (~20 year) An Earth System Modeling Environment (ESME) that combines models, data, experiments, collaborative tools and information resources in a way that fosters knowledge sharing and accelerates scientific workflow

ESMF and Application Integration Projects

NASA CAN ESMF Project Description PRODUCTS: Core framework: Software for coupling geophysical components and utilities for building components Applications: Deployment of the ESMF in 15 of the nation’s leading climate and weather models, assembly of 8 new science-motivated applications METRICS: RESOURCES and TIMELINE: $9.8M from NASA over 3 years, starting FY02 ReuseInteroperabilityEase of AdoptionPerformance 15 applications use ESMF component coupling services and 3+ utilities 8 new applications comprised of never-before coupled components 2 codes adopt ESMF with < 2% lines of code changed, or within 120 FTE-hours No more than 10% overhead in time to solution, no degradation in scaling

NASA CAN Adoption Status adoption due summer 2005

DoD Battlespace Environments Institute To develop a DoD-wide whole-earth environment which interoperates with that from other agencies: ◦Migrate core DoD models to ESMF Navy (e.g., NCOM, HYCOM, SWAN, COAMPS ™ ) Air Force (Kinematic Solar Wind and GAIM ) Army (e.g., ADCIRC, WASH123) ◦Development of tools and applications (e.g., extend ESMF to support unstructured grids and nesting) ◦Coupled applications: Air/ocean, air/ocean/ice, air/ocean/groundwater, air/space-weather ◦Thorough testing, prototyping, documentation of all components ◦$11M over six years of funding, starting FY05

NASA Modeling Analysis and Prediction Program ESMF-based computational environment for researchers investigating climate variability and change New set of climate, chemistry, and related applications coming in to ESMF NASA plan includes support for Independent Test Team Awards to be announced April 2005

ESMF Development Status Overall architecture is well-defined Components and low-level communications stable Concurrent or sequential execution, single executable Serial or parallel execution (mpiuni stub library) Logically rectangular grids with regular and arbitrary distributions can be represented and regridded On-line parallel regridding (bilinear, 1 st order conservative) completed and optimized Other parallel methods, e.g. halo, redistribution, low-level comms implemented Utilities such as time manager, logging, and configuration manager usable and adding features Virtual machine with hooks for load balancing implemented Fortran interfaces and complete documentation, some C++ interfaces

ESMF Platform Support IBM AIX (32 and 64 bit addressing)IBM AIX SGI IRIX64 (32 and 64 bit addressing)SGI IRIX64 Compaq OSF1 (64 bit addressing)Compaq OSF1 Linux Intel (32 and 64 bit addressing, with mpich and lam)Linux Intel Linux PGI (32 bit addressing, with mpich)Linux PGI Linux NAG (32 bit addressing, with mpich)Linux NAG Linux Absoft (32 bit addressing, with mpich)Linux Absoft Linux Lahey (32 bit addressing, with mpich)Linux Lahey Mac XLF (32 bit addressing, with lam)Mac XLF Mac Absoft (32 bit addressing, with lam)Mac Absoft SGI Altix (64 bit addressing)SGI Altix Cray X1 (64 bit addressing)Cray X1 These are the platforms and options that we test on nightly. Other options are provided in the ESMF build system.

Some Metrics … Core Implementation Team at NCAR currently has ◦2 FTE testers ◦1/2 FTE performance analyst ◦5 FTE developers ◦1 FTE admin/web support ◦1 manager Anticipate sustained level of staff for next 5-6 years Test suite currently consists of ◦~1250 unit tests ◦~15 system tests ◦~40 examples runs every night on ~12 platforms, ~ 40 configurations ~284 ESMF interfaces implemented, ~266 fully or partially tested, ~93% fully or partially tested – list of untested methods on ESMF website ~150,000 SLOC ~1000 downloads

Open Source Development Open source license (GPL) Open source environment (SourceForge) Open repositories: web-browsable CVS repositories accessible from the ESMF website ◦for source code ◦for contributions (currently porting contributions and performance testing) Open development priorities and schedule: priorities set by users (previously during meetings and telecons, transition to new Change Review Board), web- browsable task lists Open communication: frequent telecons and mailing list discussions Open testing: all tests are bundled with the ESMF distribution and can be run by users Open port status: results of nightly tests on many platforms are web-browsable Open metrics: test coverage, lines of code, requirements status are updated regularly and are web-browsable

ESMF Component Overhead Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall Run on NCAR IBM Runs done by JPL staff, confirmed by NCEP developers

ESMF Current Challenges Process and techniques for software requirements analysis, design and interface review with a VERY large distributed community (installation of commercial software (DOORS) for requirements archival and tracking, ties to testing and tasking) Transition to multi-agency organizational structure Design strategy for advanced grids and regridding; science and math hires to implement new grids and regridding methods Continued improvement of development and test processes Clear, complete, carefully edited documentation and training program materials New collaborations for technical and domain extension

Planned ESMF Extensions 1.Looser couplings: support for multiple executable and Grid-enabled versions of ESMF 2.Support for representing, partitioning, communicating with, and regridding unstructured grids and semi-structured grids 3.Support for advanced I/O, including I/O support for both logically rectangular, semi-structured, and unstructured grids, asynchronous I/O, checkpoint/restart, and multiple archival mechanisms (e.g. NetCDF, HDF5, binary, etc.) 4.Advanced support for data assimilation systems, including data structures for observational data and adjoints for ESMF methods 5.Support for nested, moving grids and adaptive grids 6.Support for regridding in three dimensions and between different coordinate systems 7.Advanced optimization and load balancing

Extension Process Meaningful agreement for Earth/space frameworks to collaborate and interoperate (i.e., includes resources, staged goals, timeline) Collective analysis and archival of requirements for space weather applications Collective and ongoing design, review, and testing Participation in joint meetings and events (e.g. GEM, SWW, ESMF Annual Meeting (July 20-22, Cambridge, MA) Shared tools?