1. Computational Design of the CCSM Next Generation Coupler Tom Bettge Tony Craig Brian Kauffman National Center for Atmospheric Research Boulder, Colorado USA Draft Presentation: The Fourth International Workshop on Next Generation Climate Models for Advanced High Performance Computing Facilities, March 12-14, 2002, Boulder, Colorado, USA.

2. Background 1: Current Coupler Shortcomings CSM1 load balancing Parallel Climate Model (PCM) scalability Flexibility Extensibility Portability? –CSM1 designed primarily for vector architectures –PCM designed primarily for distributed memory single/multiprocessor clusters Code reuse: poor

3. Background 2: Future Coupler Design New CCSM Coupler: Next Generation Coupler (NGC) DOE/NSF Proposal –Performance-Portable Extendible Coupler scientific requirements computational requirements –Formal Software Development Process isolate requirements (Spring 2000) state a design (abstractions, code reuse, etc.) prototyping code construction testing and validation www.ccsm.ucar.edu/models/cpl6

4. NGC Requirements: Scientific Functionality Provide overall control of coupled system Flexible and extensible: new components, new exchange information Flexible coupling frequencies Process split or time split advancement Allow computations –compute interfacial fluxes –mapping between grids –merging of information –temporal accumulation and averaging –diagnostics Process data on a variety of grids

5. NGC Requirements: Computational Functionality Portable Language: F90 or F90 interfaces Parallelism –pure shared memory –pure message passing –hybrid Flexible coupler/model interface Sequential or concurrent model execution Build as single or multiple executables I/O capability Error handling

6. NGC Design: Considerations high level command/control lower level functions --> suggests a layered design strategy concurrent or sequential execution flexibility generalized error handling --> suggests a library core of functions generalized grid representations extensibility --> suggests internal derived data types

7. Next Generation Coupler Formal Design MCT* MPH** *Model Coupling Toolkit ** Multi-Component Handshaking Library cplx (cpl6)

8. MPEU – Message Passing Env Utility From NASA DAO, provides the following services: F90 module-style access to MPI Portable/Flexible definition of types Support for multiprocessor stdout/stderr Error handling / shutdown Timing/Load balance monitoring tools Sorting Tools Support for basic derived types on which low-level classes in the MCT are built Code used as scaffolding, but utilities might fit well in CCSM.

9. MCT - Model Coupling Toolkit Major Attributes: –Maintains model decomposition descriptors (e.g., global to local indexing) –Inter- and intra- component communications and parallel data transfer (routing) –Flexible, extensible, indexible field storage –Time averaging and accumulation –Regridding (via sparse matrix vector multiply) The MCT eases the construction of coupler computational cores and component-coupler interfaces. www-unix.mcs.anl.gov/acpi/mct

10. MPH - Multi-Component Handshaking Library General Features –built on MPI –establishes MPI communicator for each component –performs component name registration –allows resource allocation for each component –supports different execution modes www.nersc.gov/research/SCG/acpi/MPH

11. cplx Architecture

12. Component Model/Coupler Interface Use coupler interface modules: use module ngc_interface_mod.F90 use module ngc_fields_mod.F90 Compile/link shared code (cpl6 modules, MCT, MPH) with model. Calls: ngc_init( modelID, localComm ) ngc_sendInit( modelID, flags(:), buffer(:,:) ) ngc_recvInit( modelID, flags(:), buffer(:,:) ) ngc_send( modelID, flags(:), buffer(:,:) ) ngc_recv( modelID, flags(:), buffer(:,:) ) ngc_sendFin( modelID, flags(:), buffer(:,:) ) ngc_finalize( modelID )

13. cpl6 Implementation Initial cpl6 - same interactions as cpl5 Concurrent component execution Multiple executables Message passing will be distributed parallel Matrix multiply will be distributed parallel

14. cpl6 Coupling Procedure/Issues multiple executable, concurrent execution (CCSM2)

15. Coupled System Processor Layout PCM CCSM2 Target? CSM1

16. Schedule and FAQ cpl6 prototype – exists cpl6beta – June 2002 cpl6 – Fall 2002 Will cpl6 be the coupler in the May 2002 CCSM2 release? No. The first release of CCSM2 will contain cpl5. Will cpl6 offer new functionality over cpl5? No. For the most part, cpl6 replicates cpl5 functionality. However, the new coupling framework provides the tools to easily build differing instantiations of a “coupler”. cpl6 is one such instantiation (e.g.., cpl6 is distributed parallel)

17. cpl [next] cpl[next] charter? –user community input to whom? (no coupler WG) –process for introducing/requesting/prioritizing new features features? –“on the fly” SCRIP (multiple options) –CCSM single binary, sequential executable cpl – cpl suitable for cluster of vector machines –concept of one model, multiple grids?