1. Nesting Algorithm Development ROMS 3.6 -r605 ROMS 3.6 -r605 ROMS 3.7 -rXXX ROMS 3.7 -rXXX NOAA Headquarters Washington, D.C February 28, 2012 Hernan G. Arango IMCS, Rutgers University

2. Acknowledgements Andrew M. Moore U. California Santa Cruz Adjoint-Based Algorithms Alexander F. Shchepetkin U. California Los Angeles Nonlinear Kernel David J. Robertson IMCS, Rutgers University Cyber Infrastructure Kate S. Hedstrom U. Alaska, Fairbanks User Community Forum John C. Warner USGS, WHOI Sediment Transport, Nesting

3. Development Strategy The nesting option is implemented in three phases: Phase I, released on April 2011 as version 3.5: Change ROMS numerical kernels (NLM, TLM, RPM, ADM) to allow different horizontal i- and j-ranges in DO-loops to permit operations on various nested grid classes (refinement, mosaics, and composite) and nesting layers (refinement and composite combinations). This facilitates the computation of any horizontal operator (advection, diffusion, gradient, etc.) in the nesting overlap regions and avoids the need for cumbersome lateral boundary conditions on the model variables and their associated flux/gradient values. The advantage of this approach is that it is generic to any discrete horizontal operator. The overlap region is an extended section of the grid that overlays an adjacent grid. The strategy is to compute the full horizontal operator at the contact points between nested grids instead of specifying boundary conditions.

4. Tile I- and J-Ranges Istr = BOUNDS(ng) % Istr(tile) IstrB= BOUNDS(ng) % IstrB(tile) IstrM= BOUNDS(ng) % IstrM(tile) IstrP= BOUNDS(ng) % IstrP(tile) IstrR= BOUNDS(ng) % IstrR(tile) IstrT= BOUNDS(ng) % IstrT(tile) IstrU= BOUNDS(ng) % IstrU(tile) Iend= BOUNDS(ng) % Iend(tile) IendB= BOUNDS(ng) % IendB(tile) IendP= BOUNDS(ng) % IendP(tile) IendR= BOUNDS(ng) % IendR(tile) IendT= BOUNDS(ng) % IendT(tile) Jstr= BOUNDS(ng) % Jstr(tile) JstrB= BOUNDS(ng) % JstrB(tile) JstrM= BOUNDS(ng) % JstrM(tile) JstrP= BOUNDS(ng) % JstrP(tile) JstrR= BOUNDS(ng) % JstrR(tile) JstrT= BOUNDS(ng) % JstrT(tile) JstrV= BOUNDS(ng) % JstrV(tile) Jend= BOUNDS(ng) % Jend(tile) JendB= BOUNDS(ng) % JendB(tile) JendP= BOUNDS(ng) % JendP(tile) JendR= BOUNDS(ng) % JendR(tile) JendT= BOUNDS(ng) % JendT(tile) Istrm3= BOUNDS(ng) % Istrm3(tile)Istr-3 Istrm2= BOUNDS(ng) % Istrm2(tile)Istr-2 Istrm1= BOUNDS(ng) % Istrm1(tile)Istr-1 IstrUm2= BOUNDS(ng) % IstrUm2(tile)IstrU-2 IstrUm1= BOUNDS(ng) % IstrUm1(tile)IstrU-1 Iendp1= BOUNDS(ng) % Iendp1(tile)Iend+1 Iendp2= BOUNDS(ng) % Iendp2(tile)Iend+2 Iendp2i= BOUNDS(ng) % Iendp2i(tile)Iend+2 interior Iendp3= BOUNDS(ng) % Iendp3(tile)Iend+3 Jstrm3= BOUNDS(ng) % Jstrm3(tile)Jstr-3 Jstrm2= BOUNDS(ng) % Jstrm2(tile)Jstr-2 Jstrm1= BOUNDS(ng) % Jstrm1(tile)Jstr-1 JstrVm2= BOUNDS(ng) % JstrVm2(tile)JstrV-2 JstrVm1= BOUNDS(ng) % JstrVm1(tile)JstrV-1 Jendp1= BOUNDS(ng) % Jendp1(tile)Jend+1 Jendp2= BOUNDS(ng) % Jendp2(tile)Jend+2 Jendp2i= BOUNDS(ng) % Jendp2i(tile)Jend+2 interior Jendp3= BOUNDS(ng) % Jendp3(tile)Jend+3 Suffix: R:tile RHO-pointsB:Boundary tile RHO- and V-points U:tile U-pointsM:Boundary tile PSI- and U-points V:tile V-pointsP:Nesting PSI-, U-, and V-points T:Nesting RHO-points

5. If not nesting grids, the additional boundary tile indices associated with nesting are set to: IstrT = IstrR full range, startingI-direction (RHO-point) IendT = IendR full range, endingI-direction (RHO-point) JstrT = JstrR full range, startingJ-direction (RHO-point) JendT = JendR full range, endingJ-direction (RHO-point) IstrP = Istr full range, startingI-direction (PSI-, U-point) IendP = Iend full range, endingI-direction (PSI-point) JstrP = Jstr full range, startingJ-direction (PSI-, V-point) JendP = Jend full range, endingJ-direction (PSI-point) IstrB = Istr interior range, startingI-direction (RHO-, V-point) IendB = Iend interior range, endingI-direction (RHO-, V-point) JstrB = Jstr interior range, startingJ-direction (RHO-, U-point) JendB = Jend interior range, endingJ-direction (RHO-, U-point) IstrM = IstrU interior range, startingI-direction (PSI-, U-point) JstrM = JstrV interior range, startingJ-direction (PSI-, V-point) Boundary Tile Indices

6. Boundary Tile Indices Locations

7. mod_param.F

8. get_bounds.F

9. Development Strategy Phase II, released on September 2011 as version 3.6: Major overhaul of ROMS lateral boundary conditions. All C-preprocessing options were eliminated and replaced with logical switches (LBC structure) that depend on the nested grid, if any. This facilitates, in a generic way, the processing or not of lateral boundary conditions in applications with nested grids. In nesting applications, the values at the lateral boundary points are computed directly in the overlap region by the numerical kernel. The logical switches allow different lateral boundary conditions types between active (temperature and salinity) and passive (biology, sediment, inert, etc.) tracers. The lateral boundary condition switches for each state variable and boundary edge are now specified in ROMS input script file, ocean.in.

10. Lateral Boundary Conditions Structure TYPE T_LBC logical :: acquireprocess lateral boundary data logical :: Chapman logical :: clamped logical :: closed logical :: Flather logical :: gradient logical :: nested logical :: nudging logical :: periodic logical :: radiation logical :: reduced END TYPE T_LBC TYPE (T_LBC), allocatable :: LBC(:,:,:) For example, for free-surface gradient boundary conditions we have: LBC(iwest,isFsur, ng) % gradient LBC(ieast,isFsur, ng) % gradient LBC(isouth,isFsur, ng) % gradient LBC(inorth,isFsur, ng) % gradient

11. u2dbc_im.F

12. For Example, in zetabc.F the western boundary conditions are: IF ( DOMAIN (ng) % Western_Edge(tile) ) THEN IF ( LBC (iwest, isFsur, ng) % radiation ) THEN … ELSE IF ( LBC (iwest, isFsur, ng) % Chapman ) THEN … ELSE IF ( LBC (iwest, isFsur, ng) % clamped ) THEN … ELSE IF ( LBC (iwest, isFsur, ng) % gradient ) THEN … ELSE IF ( LBC (iwest, isFsur, ng) % closed ) THEN DO j = Jstr, Jend IF ( LBC_apply (ng) % west ( j ) ) THEN ! Allows both specified and zeta ( Istr-1, j, kout ) = zeta ( Istr, j, kout ) ! nested conditions END IF END DO END IF Lateral Boundary Conditions Code

13. Standard Input File: ocean.in

14. Development Strategy Phase III, to be released soon as version 3.7: includes the nesting calls to ROMS main time-stepping routines, main2d and main3d. The concept of nesting layers is introduced to allow applications with both composite grids and refinement grids. Several new routines are added to process the information that it is required in the overlap region, what information needs to be exchanged from/to another grid, and when to exchange it. In mosaic and composed grids, the information is exchanged between each sub-time step call in main2d or main3d. For example, the data donor grid and the mosaic/composite grids need to sub-time step the 2D momentum equations before any of them start solving and coupling the 3D momentum equations. In refinement grids, the information at the contact points is processed at the beginning of the full time-step layer. The exchange between data donor and refinement grids is two-way.

15. main_3d.F

16. nesting.F

17. Nesting Configuration Types

18. Composite Grids CaseGridLayerDonorWestSouthEastNorth a 1122LBC 211 1 b 112LBC,2LBC 211 1 c 112LBC,2LBC 211 LBC,11LBC Mosaic Class: Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 Composite Class: Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 Composite Overlap Class: Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 abc

19. Composite Grids: Complex Estuary GridLayerDonorWestSouthEastNorth 112, 3, 4, 5, 6LBC, 2, 3LBCLBC, 4, 5, 6LBC 211 1 311 1 4111 5111 6111 Composite Estuary Class: Ngrids = 6 NestLayers = 1 GridsInLayer = 6 Ncontact = 10

20. Refinement Grids CaseGridLayerDonorWestSouthEastNorth a 11n/aLBC 2211111 b 11n/aLBC 2211111 3211111 4322222 Refinement Class: Ngrids = 2 NestLayers = 2 GridsInLayer = 1 1 Ncontact = 2 Multi-Refinement Class: Ngrids = 4 NestLayers = 3 GridsInLayer = 1 2 1 Ncontact = 6 a b

21. Composite-Refinement Grid Combination GridLayerDonorWestSouthEastNorth 112,3LBC,2LBC 211 1 3211111 Composite-Refinement Class: Ngrids = 3 NestLayers = 2 GridsInLayer = 2 1 Ncontact = 4

22. (Rancic and Purser 1996) Cubed-Sphere grid Regular grid mosaic Refined grid mosaic Mosaics Grids

23. Tripolar Grid (Murray 1996, Griffies et al 2004)

24. Monterrey Bay Nested Grids 3:1 Ratio 5:1 Ratio

25. Tripolar Grid: ORCA R025 (1/4 o at the Equator, 1442x1021 points) Orthographic Projection Stereographic Projection

26. Contact Regions and Contact Points

27. integer :: Ncontact total number of contact regions TYPE T_NGC logical :: coincident coincident donor and receiver points, p=q=0 logical :: interpolateperform vertical interpolation integer :: donor_griddata donor grid number integer :: receiver_griddata receiver grid number integer :: Npoints integer, pointer :: Idg (:) donor grid, cell I-left index integer, pointer :: Jdg (:) donor grid, cell J-bottom index integer, pointer :: Kdg (:, :)donor grid, cell K-index integer, pointer :: Irg (:)receiver grid, I-contact point integer, pointer :: Jrg (:)receiver grid, J-contact point real(r8), pointer :: Hweight (:, :) horizontal weights real(r8), pointer :: Vweight(:, :, :)vertical weights END TYPE T_NGC TYPE (T_NGC), allocatable :: Rcontact (:)RHO-points TYPE (T_NGC), allocatable :: Ucontact (:) U-points TYPE (T_NGC), allocatable :: Vcontact (:) V-points Contact Points Structure

28. Contact Points Interpolation Hweight (1, :) = (1 - p) * (1 - q) Hweight (2, :) = p * (1 - q) Hweight (3, :) = p * q Hweight (4, :) = (1 - p) * q Value (Irg, Jrg) = Hweight (1,:) * F2d(Idg,Jdg )+ Hweight (2,:) * F2d(Idg+1,Jdg )+ Hweight (3,:) * F2d(Idg+1,Jdg+1)+ Hweight (4,:) * F2d(Idg,Jdg+1) If coincident contact points between data donor and data receiver grids, p = q = 0.0, Hweight (1, :) = 1.0Value (Irg, Jrg) = Hweight (1, :) * F2d( Idg, Jdg ) Hweight (3, :) = 0.0 Hweight (4, :) = 0.0 Hweight (5, :) = 0.0 3 (Idg+1, Jdg+1, Kdg-1) 2 4 (Idg, Jdg, Kdg-1) 1 Irg Jrg q 1- q 1- p p 3 Kdg-1 7 Kdg 5 6 4 12 8 5 Suffix: dg = donor grid rg = receiver grid

29. Contact Regions Metadata: Additional NetCDF File

30. Nested Grids: Mosaic Class Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2

31. Nested Grids: Composite Class Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 Donor Grid = blue Receiver Grid = red

32. Nested Grids: Composite Class Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 Donor Grid = red Receiver Grid = blue

33. Nested Grids: Composite Overlap Class Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 Donor Grid = blue Receiver Grid = red

34. Nested Grids: Composite Overlap Class Ngrids = 2 NestLayers = 1 GridsInLayer = 2 Ncontact = 2 Donor Grid = red Receiver Grid = blue

35. Nested Grids: Composite Estuary Class Ngrids = 6 NestLayers = 1 GridsInLayer = 6 Ncontact = 10 Donor Grids = red Receiver Grid = blue

36. Nested Grids: Composite Estuary Class Ngrids = 6 NestLayers = 1 GridsInLayer = 6 Ncontact = 10 Donor Grid = blue Receiver Grids = red

37. Nested Grids: Refinement Class Ngrids = 2 NestLayers = 2 GridsInLayer = 1 1 Ncontact = 2 Donor Grid = blue Receiver Grid = red

38. Nested Grids: Refinement Class Ngrids = 2 NestLayers = 2 GridsInLayer = 1 1 Ncontact = 2 Donor Grid = red Receiver Grid = blue

39. Nested Grids: Multi-Refinement Class Ngrids = 4 NestLayers = 3 GridsInLayer = 1 2 1 Ncontact = 6 Donor Grid = blue Receiver Grid = red Ngrids = 4 NestLayers = 3 GridsInLayer = 1 2 1 Ncontact = 6 Donor Grid = red Receiver Grid = green

40. Nested Grids: Multi-Refinement Class Ngrids = 4 NestLayers = 3 GridsInLayer = 1 2 1 Ncontact = 6 Donor Grid = red Receiver Grid = blue Ngrids = 4 NestLayers = 3 GridsInLayer = 1 2 1 Ncontact = 6 Donor Grid = green Receiver Grid = red

41. Nested Grids: Composite-Refinement Class Ngrids = 3 NestLayers = 2 GridsInLayer = 2 1 Ncontact = 4 Donor Grid = blue Receiver Grids = red

42. Nested Grids: Composite-Refinement Class Ngrids = 3 NestLayers = 2 GridsInLayer = 2 1 Ncontact = 4 Donor Grids = red Receiver Grid = blue

43. Python Based GUI

46. Remarks ROMS nesting capabilities are unique and allows complex estuary and coastal configurations with unlimited number of composite and refined grids Coincident composite and mosaic grids produce identical solutions when compared to one large continuous grid Complete overhaul of ROMS numerical kernels: NLM, TLM, RPM, and ADM Development divided in three phases. Phase I released as ROMS 3.5 on April 25, 2011. Phase II released as ROMS 3.6 on Sep 23, 2011. The final Phase III will be released as ROMS 3.7 soon… A Python-based GUI is currently under development to facilitate building input scripts in complex configurations We need to start looking to more sophisticated grid generation tools

47. Warner, J.C., W.R. Geyer, and H.G. Arango, 2010: Using composite grid approach in complex coastal domain to estimate estuarine residence time, Computer and Geosciences, 36, 921-935, doi:10.1016/j.cageo.2009.11.008. Reference