1. www.esmf.ucar.edu Cecelia DeLuca, Don Stark, Chris Hill Arctic System Model Workshop May 20, 2008 Earth System Modeling Framework

2. www.esmf.ucar.edu Outline About ESMF Best Practices, On and Off List Governance Personal History Conclusion

3. www.esmf.ucar.edu About ESMF ESMF provides component wrappers with standard interfaces, a set of data structures for data exchanges, and customizable drivers. ESMF provides common utilities, such as data communications, regridding, time management, configuration, and message logging. Goals are component interoperability and software reuse.

4. www.esmf.ucar.edu Each box is an ESMF component ESMF State objects carry data between components Every component (including couplers) has a standard interface to facilitate exchanges The ESMF architecture enables the assembly of many different systems ESMF Application Example GEOS-5 Atmospheric General Circulation Model Application Example

5. www.esmf.ucar.edu Standardization of Component APIs There are only three ESMF component methods: Initialize, Run, and Finalize (I/R/F) Users create a Component by assigning their user code I/R/F methods to an ESMF Component type The ESMF Component calls back into the specific user-assigned methods I/R/F methods cascade down the tree Small set of standard arguments: call ESMF_CompRun (myComp, importState, exportState, clock, phase, blockingFlag, rc)

6. www.esmf.ucar.edu Standardization of Data Structures ESMF State data structures contain all data exchanged between components Flexibility in data representation - States can contain lists of varied data structures, including Arrays, ArrayBundles, Fields, FieldBundles, and other States The ESMF philosophy is to constrain the number of component methods and their arguments, but retain flexibility in the data structures used in inter-component exchanges

7. www.esmf.ucar.edu ESMF Adoption - PARSSSE P repare user code Split user code into initialize, run and finalize methods and decide on components and coupling fields and control flow A dapt data structures Wrap native data structures either as ◦ESMF Arrays (represent data in index space only) Regrid via user-supplied weights + sparse matrix multiply ◦ESMF Field objects (Arrays + coordinates and metadata) Regrid via on-line weight generation Pack Arrays or Fields into States. Wrap time information in ESMF Clocks. R egister user methods Attach user code initialize, run and finalize methods to ESMF Components through a registration process S chedule, S ynchronize, and S end data between components Write couplers using ESMF redistribution, sparse matrix multiply, regridding, or user-specified transformations E xecute the application Run components using an ESMF driver

8. www.esmf.ucar.edu ESMF Earth Science Components & Models HAF GAIM

9. www.esmf.ucar.edu ESMF Timeline First phase started in 2002 with NASA funding Second phase began in 2005 with a transition to multi-agency support and management – NASA, NOAA, NSF, and DoD sponsors ESMF component interfaces are the basis of programs at multiple agencies: ◦DoD Battlespace Environments Institute (BEI) ◦NOAA National Environmental Modeling System (NEMS) ◦NASA Modeling Analysis and Prediction Program for Climate Variability and Change (MAP) ◦Community Sediment Transport Model (ONR), others… In operational use since August 2006, initially at the National Weather Service, ongoing transition of other ESMF coupled systems to operations Scope of current work will be completed in 2010 Ongoing diversified funding expected

10. www.esmf.ucar.edu Release Path 2002 2003 2004 2005 2006 2007 2008 2009 2010 ESMF v1 Prototype ESMF v2 Components, VM and Utils ESMF_GridCompRun() ESMF v3 Index Space Operations ESMF_ArraySparseMatMul() ESMF v4 Grid Operations ESMF_GridCreate() ESMF_FieldRegrid() ESMFv5 Standardization Build, init, data types, error handling, … Public release ESMF v3.1.0r Full system prototype Building bottom-up Standardization

11. www.esmf.ucar.edu ESMF Package Description, the Basics Open source, open development (browsable repository) Components can run concurrently, sequentially or in mixed mode Serial or parallel Single or multiple executable or combinations Shared or distributed memory or hybrid Support for model ensembles, including execution of multiple instances in the same address space 2000+ unit tests, system tests, and examples regression tested nightly on 26+ platform/compiler combinations2000+ unit tests, system tests, and examples regression tested nightly on 26+ platform/compiler combinations Reference Manual, Users Guide and the examples therein updated automatically with code changesReference Manual

12. www.esmf.ucar.edu ESMF Package Description, Coupling Data transformations can be executed within a coupler component, or arranged in a coupler component and executed directly between model components Coupling can be done in index space or physical space Performance: < 5% overhead in time to solution vs customized native approaches, highly scalable in performance and memory (performance reports online)performance reports online

13. www.esmf.ucar.edu Upcoming Releases Public Release 3.1.0r, May 2008 Main capabilities added since last public release - 2.2.2rp3, December 2006: ◦General grid and data representation and parallel regridding support through an Array index space layer (user supplied interpolation weights) – logically rectangular grids, multi-patch grids, unstructured grids ◦Representation of rectilinear and curvilinear grids in a Grid class (preparation for framework-generated interpolation weights) ◦New ports including gfortran and Solaris Compact, simpler interface: cut number of public data handling methods in half (304 -> 147, with caveats)

14. www.esmf.ucar.edu Upcoming Releases (cont.) Internal Release 3.1.1, June 2008 Initial implementation of observational data streams Generation of regridding interpolation weights for logically rectangular grids, bilinear and higher order method Attribute class can store standardized metadata packets and represent metadata hierarchies - e.g., State metadata includes the metadata of Fields that are stored in it Initial implementation of unstructured grid interface Patch Release 3.1.0rp*, Summer 2008 Petascale optimizations for Array and ArrayBundle sparse matrix multiply

15. www.esmf.ucar.edu Relationship to CCSM Passed CCSM Stage 1 evaluation of ESMF in May 2007, showing < 5% overhead, comparable memory use, no change to component internals. Result: creation of cpl7 based on ESMF design Currently working on Stage 2 evaluation: Creation of an ESMF version of cpl7 called ESMCoupler Expected advantages: ◦Greater interoperability with other components ◦More flexible architecture that doesn’t rely on a single central coupler ◦On-line regridding ◦Integrated handling of metadata

16. www.esmf.ucar.edu Team Composition Team composition (blue is off-site) 1.Manager (agency coordination and technical coordination) 2.Operations Manager (website, metrics, space issues, local administration) 3.Integrator/Test lead (regression testing, release management) 4.Tester for numerical methods 7 developers: 5.Systems level/porting/build 6.Low level data structures and architecture 7.Structured grids 8.Meshes 9.High level numerical data structures 10.Language interfaces 11.Utilities including calendaring 12.Metadata and attributes External and related 13.External: ½ FTE performance testing 14.Related: FTE for component distribution portal

17. www.esmf.ucar.edu Distributed Development GOAL - Everybody on the team has access to all information, current and past Archived email list where all development correspondence gets cc’d Frequent telecons with minutes Web browsable repository, mail summary on check-ins Daily archived test results Monthly archived metrics Public archived trackers (bugs, feature requests, support requests, etc.)

18. www.esmf.ucar.edu ESMF Governance Executive Board Strategic Direction Organizational Changes Board Appointments Interagency Working Group Stakeholder Liaison Programmatic Assessment & Feedback Advisory Board External Projects Coordination General Guidance & Evaluation Functionality Change Requests Joint Specification Team Requirements Definition Design and Code Reviews External Code Contributions Implementation Schedule Resource Constraints Collaborative Design Beta Testing Working Project Executive Management Reporting weekly Core Development Team Project Management Software Development Testing & Maintenance Distribution & User Support daily annuall y Change Review Board Development Priorities Release Review & Approval quarterly Multiple timescales, all staff levels Places for structured argument

19. www.esmf.ucar.edu Values Community driven development and community ownership Openness of project processes, management, code and information Correctness Commitment to a globally distributed and diverse development and customer base Simplicity Efficiency Public storage of project records and other information Engagement Web link for detail: http://www.esmf.ucar.edu/about_us/values.shtml

20. www.esmf.ucar.edu How to Get Started & Get Help Web meeting with the development team Support list – esmf_support@ucar.edu Code Examples page and FAQ Tutorials, on-line and in person Annual Meeting (Fort Lauderdale, FL May 28-30, 2008) Reference Manual and Users Guide

21. www.esmf.ucar.edu Governance Management of ESMF requires governance that recognizes social and cultural factors as well as technical factors Main objectives of governance: Enabling people to argue in a civilized, contained, constructive way Enabling people to set priorities and make decisions based on resource realities Coordinating at many levels, from developer to center director to program manager

22. www.esmf.ucar.edu Facilitating Science: Coupled Climate-Chemistry with ESMF The image shows results from a version of the GEOS-5 atmospheric general circulation model coupled to a stratospheric chemistry package (STRAT-CHEM), also developed at NASA, but independently of GEOS-5, which has now been made ESMF compliant.