1. Expanding Regridding Capabilities of the Earth System Modeling Framework Andrew Scholbrock University of Colorado – Boulder Robert Oehmke NOAA/CIRES 1

2. Outline Introduction Background Information Incorporating LibCF Regridding into ESMF Current Status Future Work 2

3. Introduction This project aims to allow regridding software that follows a proposed metadata standard (Gridspec) to be used within the Earth System Modeling Framework (ESMF) As new regridding capabilities are developed they will be automatically available through ESMF 3

4. ESMF Framework developed by a multi-agency consortium [1] Provides standard interfaces for model components Provides standard structure for transferring data between model components Provides a range of utilities to ease coupling – Data regridding – Data redistribution – Time management – Error handling 4

5. Gridspec Metadata convention developed at NOAA GFDL [2] Proposed for inclusion in the Climate and Forecast (CF) [3] conventions Gridspec represents grids as mosaics and tiles – Mosaic files contain a list of tile files and how they are connected – Tile files contain actual grid coordinate information 5 http://www.gfdl.noaa.gov/~vb/gridstd/grid stdse2.html#x4-160002.7

6. LibCF Utility library developed at Unidata Built on top of NetCDF [4] Climate and Forecast (CF) conventions LibCF [3] includes the capability to: – Manipulate Gridspec metadata on a NetCDF file – Create Gridspec files – Regrid between different Gridspec files LibCF is still in alpha release mode 6

7. Regridding Overview Climate and other models require coupling of modeling components (e.g. atmosphere, ocean) Different modeling components may require different grids – e.g. different shapes or different resolutions Regridding needs to be performed to move data between components on different grids Steps involved in regridding: – Generate interpolation weights – Apply weights to interpolate field using a sparse matrix multiply 7

8. ESMF Regridding Methods of accessing regridding: – Online Subroutine calls which calculate weights during a model run Can get weights or feed directly into ESMF sparse matrix multiply – Offline Application which generates a netCDF interpolation weight file from two netCDF grid files Computation of weights can be done in parallel Supported grids: – 2D meshes composed of triangles or quadrilaterals – 3D meshes composed of hexahedra – 2D/3D logically rectangular grids – Cubed sphere Supported interpolation methods: – Bilinear – Higher order finite element patch recovery [5][6] – Conservative (beta version) Not all combinations of the above are supported 8

9. LibCF Regridding LibCF takes Gridspec files and field data files as input to generate weights and interpolate data – If just Gridspec files are specified then only interpolation weights are generated LibCF supports regridding between mosaics of logically rectangular grid tiles (e.g. cubed sphere, single tile logically rectangular) Currently first order conservative interpolation methods are supported, and some specific bilinear cases 9

10. Combining ESMF and LibCF LibCF was introduced into the ESMF online regridding software. This was somewhat awkward because LibCF requires Gridspec files as input, but ESMF online regridding holds grid descriptions in memory. As a result, ESMF must convert its grid descriptions to Gridspec files and then have LibCF read them in. This works, but more efficient connections could be made with regridding libraries that did not require file input. 10

11. Implementation Steps: Getting Interpolation Weights 1.Write ESMF grids to NetCDF files a)Take an existing ESMF grid and get the coordinate information out of it b)Write out the coordinate information to a NetCDF file Done using ESMF I/O functionality c)Create mosaic files from the NetCDF grid Done using LibCFs make_mosaic function 2.Get interpolation weights from LibCF a)Call LibCFs regridding function through ESMF calls b)Read in LibCFs interpolation weights into ESMF c)Convert interpolation data to a form that ESMFs sparse matrix multiply can use 11

12. Implementation Steps: Weight Application Apply LibCF interpolation weights to field data using the ESMF sparse matrix multiply call Future regrid calls to updated field data can bypass the LibCF interpolation weight generation step – This holds as long as the grids involved remain the same 12

13. LibCF/ESMF Regridding Flow Write Tile Source Grid Create Mosaic Source Tile Create Weights Read Weights Weight Matrix make_mosaicfregrid Source Mosaic Remap File Source Variable Data Destination Variable Data ESMF LibCF Destination Tile Destination Grid Destination Mosaic Regrid File I/O NetCDF File ESMF Data API Call Legend: 13

14. ESMF User Interface for Regridding with LibCF Actual interface that users employ contains two steps: 1.ESMF_FieldRegridStore(): Provide source/destination grids and receive interpolation weights 2.ESMF_FieldRegrid(): Provide interpolation weights and source field data to receive interpolated field data corresponding to the destination grid This is the same sequence of calls used for pure ESMF regridding 14

15. Current Status Can read interpolation weight information generated by LibCF into ESMF sparse matrix representation and perform a regrid. Can write ESMF grid structures to Gridspec files ESMF restriction: – Writing Gridpsec files from ESMF is serial and thus restricts the scaling of the final code Intersection of LibCF/ESMF capabilities: – Only supports single tile logically rectangular grids LibCF restriction: – Only supports first order conservative remapping for this geometry 15

16. Future Work Parallel I/O for writing Gridspec files Switch to non-file based interface if LibCF develops one Allow for regridding of more complicated grids including the cubed sphere 16

17. References 1.Hill, C., C. DeLuca, V. Balaji, M. Suarez, and A. da Silva (2004). Architecture of the Earth System Modeling Framework. Computing in Science and Engineering, Volume 6, Number 1, pp. 18-28. 2.Balaji, V., Liang, Z. Gridspec: A Standard for the Description of Grids Used in Earth System Models. http://www.gfdl.noaa.gov/~vb/gridstd/gridstd.html 3.The CF Library Users Guide. http://www.unidata.ucar.edu/software/libcf/docs/libcf.html 4.Hartnett, E. and Rew, R., Experience with an enhanced netCDF data model and interface for scientific data access, 88th AMS Annual Meeting, 24 th Conference on IIPS, 2008. 5.Khoei S.A., Gharehbaghi A. R. The superconvergent patch recovery technique and data transfer operators in 3d plasticity problems. Finite Elements in Analysis and Design, 43(8), 2007. 6.Hung K.C, Gu H., Zong Z. A modified superconvergent patch recovery method and its application to large deformation problems. Finite Elements in Analysis and Design, 40(5- 6), 2004. Questions? scholbro@mail.ucar.edu 17