1. Coupling ROMS and WRF using MCT
Coupling design and implementation

2. ROMS 2.0 ROMS 2.0 – new version changed from F77 to F90/F95
explicit interfaces subject to “strong typing”
allocation is via dereferenced pointer structures
parallel framework includes shared- and distributed- memory paradigms

3. WRF-Weather Research and Forecasting
Designed for:
1-10 km horizontal grid resolution
advanced data assimilation and model physics
both research and operations
performance and maintainability
Developed at:
NCAR
NOAA – NCEP/FSL/GFDL
EPA – Atmospheric Modeling Division
University Community

4. WRF – Software design

5. The Coupler
WRF and ROMS are coupled using the Model Coupling Toolkit (MCT) developed at Argonne National Labs
MCT handles the passing of variables between the ocean and atmosphere models, as well as regridding and time averaging

6. The Model Coupling Toolkit
Cons
Component model processing element (PE) pool sizes remain constant
Components can exchange only real and integer data as groups of vectors
Pros
Any number of components (ocean, atmos, ice, wave spray, etc)
Any decomposition
Any number of processors-per component
local re-gridding supported by sparse-matrix multiplication
The MCT user can supply:
Consistent numbering schemes for grid points
Integer ID for each component
An MPI communicator for each component
Interpolation Matrix Elements

7. Parallel data passing
Each component model in the MCT framework passes the coupler
information about the decomposition of the data arrays in the form of a Global Segment Map
This allows for parallel data transfer, and the ability to regrid from
the atmosphere grid to the ocean on local processors.
A sample decomposition, and resulting Global Segment Map is shown in the next slide.

8. Numbering of gridpoints Processor Decomposition
GlobalSegMap Examples 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2
atmosphere
Pe_loc start length 1
Total number
of segments = 3
Pe_loc start length
Processor Decomposition 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2 3
Total number
of segments = 10
ocean 1

9. Coupling through I/O API
WRF I/O API abstracted to allow changing packages (netCDF, HDF5, etc)
passing 2-D fields at the air-sea interface equivalent to file I/O
standard interface allows
easy interchange of components
easy interchange of variables passed
ROMS changes (all CPP “switches”)
new module “mod_io_couple.F”
ocean.F subroutinized – takes communicator as argument, passed to distribute.F
communicator pass to distribute.F, MPI initialization and finalization done by coupler
input file now handles tiling

10. Variables Passed Atmospheric Model (WRF) Oceanic Model (ROMS)
SST - Sea Surface Temp
t - Wind Stress
Q - heat (short wave and
long wave)
E - Evaporation
P - Precipitation
QLW
QSENS
QLATENT t
QSW
E-P
SST
Oceanic Model
(ROMS)

11. Alpha coupling --The first runs
The initial coupling took a sample test problem from WRF
that integrates the evolution of a supercell. ROMS and WRF
were run on the same grid so no sparse-matrix multiplication
was needed, and the fast dynamics of the 3 hour simulation
meant that ROMS and WRF could be run at similar timestep
sizes to eliminate the need for time averaging.
ROMS was given an initial stratification typical of summertime
off the Florida coast, where supercells have been observed, and
was forced only by the WRF-generated surface winds

12. Initial Model Results Updraft cells (yellow/white) drove
strong convergent surface winds
(white arrows) resulting in
strong (2-3 m/s) surface currents.
Surface height (color slice)
variations of a half-meter over
20 km were observed, and waves
propagate along the thermocline
(shown in blue).

13. Initial Model Results, cont’d

14. Future Work
Currently only SST and Winds are passed between models - other variables need to be passed and implemented into model forcing
Allow models to run on separate grids (regridding with MCT)
Time averaging of fields using MCT accumulator needs to be implemented
Coupling across the TeraGrid (next slide)

15. Coupled Modeling Across the TeraGrid
NCSA IA-32 Cluster (Myrinet)
PSC Alpha Cluster (Quadrics)
Wind Stress
ROMS
WRF
SST
In a collaborative effort between the NOAA PMEL and FSL laboratories, NCAR and Argonne, a version of the coupled WRF/ROMS model is being developed in which the ROMS component runs on one TeraGrid machine and the WRF component runs on a second TeraGrid machine. Intra-component communication occurs over, for example, Myrinet or Quadrics while inter-component communication (exchange of boundary conditions) will occur over the TeraGrid fiber-optic backbone. MPICH-G2, a globus-enabled version of MPI will provide the communication library used to implement all communication. The coupled model is divided into components using the Lawrence Berkeley Laboratory Multi-Program Component Handshaking (MPH) software. Parallel re-gridding of boundary conditions exchanged between the two models will be implemented using the Argonne Model Coupling Toolkit (MCT).

16. Questions?

17. Abstract
WRF/ROMS coupling design and implementation using the Model Coupling Toolkit
Rob Jacob, Dale Haidvogel, Al Hermann, John Michalakes, Christopher Moore Dan Schaffer
WRF and ROMS are coupled using the Model Coupling Toolkit (MCT) developed at Argonne National Laboratory. MCT is a Fortran90 library built on top of MPI with data types and methods that simplify the construction of distributed-memory parallel couplers. The coupler design itself is parallel, avoiding bottlenecks by allowing for parallel exchange of fields between models on different grids, and time averaging over the coupling period. The coupling is wired into the mediation layer in WRF, and into the I/O layer in ROMS.
The initial coupling passes wind-stress from WRF to ROMS and sea surface temperature from ROMS to WRF. Longwave and shortwave heat and evaporation/precipitation will soon be added as variables passed. A simulation is run with WRF creating a supercell, and ROMS integrating the oceanic response. WRF creates the typical “hook-return” convective updraft usually seen in storms that generate supercells, as well as high precipitation and updraft-splitting. The ROMS response shows upwelling/downwelling patterns centered on the supercell updraft location, and oceanic circulation that mimics that of measured oceanic response to offshore supercell storms.
Future work includes developing an API that will allow coupling and I/O using MCT and HDF5, and utilizing this API in both WRF and ROMS, as well as coupling across the TeraGrid using MPICH-G2.

18. MCT Distribution Architecture{
High-level MCT classes
Model Coupling Toolkit
Low-level MCT classes
Message-Passing
Environment Utilities
(MPEU)