1. Developments in the dynamical core of the global semi- Lagrangian SL-AV model Mikhail Tolstykh, Vladimir Shashkin Institute of Numerical Mathematics, Russian Academy of Sciences and Hydrometcentre of Russia With contribution by Vassili Mizyak

2. Outline SL-AV model Mass-conservative shallow-water version on the reduced lat-lon grid - results of standard SW tests New solvers for U-V reconstruction and Helmholtz problem

3. SL-AV model. General features Global finite-difference semi-Lagrangian semi- implicit dynamical core of own development + parameterizations of subgrid-scale processes from French model ARPEGE/ALADIN. Main Russian global operational model for medium- range and seasonal forecasts. So far, regular lat-lon grid is used. Reduced lat-lon grid in the shallow water version

4. SL-AV model. Distinct features of dynamical core Vorticity-divergence formulation on the unstaggered grid, wide use of 4 th order finite differences, usual and compact (Tolstykh, JCP 2002). Semi-implicit solver, reconstruction of U and V, horizontal diffusion are carried out in Fourier space in longitude. Version with variable resolution in latitude (quasioperational) Scales up to 100-300 cores, MPI+OpenMP

5. Reduced lat-lon grid

6. Reduced grid in the SL-AV model A part of calculations is carried out in space of Fourier coefficients in longitude. The semi-Lagrangian advection is used (no nonlinear advective terms) => Necessary latitudinal derivatives (i.e. geopotential gradient) can be calculated in Fourier space. Number of gridpoints at each latitudinal circle should be suitable for FFT. Currently, implemented in the shallow-water version (standard and mass-conservative versions).

7. Motivation for a new version of dynamical core Future use for climate simulations Future horizontal resolution of O(1-10km) => Mass conservation More scalability => new algorithms

8. Next version of the SL-AV dynamical core Mass conservative. More uniform resolution on the sphere – reduced lat-lon grid. More scalable - abandon data transpositions - new parallel algorithms to solve block- tridiagonal linear systems.

9. Mass-conservative shallow-water version of SL-AV Mass-conservative semi-Lagrangian advection scheme on the sphere on the reduced lat-lon grid. Mass-conservative semi-implicit time integration scheme. This implies a conservative Helmholtz problem solver on the sphere.

10. Mass-conservative semi-Lagrangian advection scheme on the sphere on the reduced grid Global conservative cascade scheme (CCS) was implemented for SL advection following “Efficient Conservative Global Transport Schemes for Climate and Atmospheric Chemistry Models” R.D. Nair et al. MWR 2002. (Meridional CFL number ≤ 1) Some changes were introduced for using CCS on the reduced grid. Example of the reduced grid (region) Stars – grid points Regions bounded by dashed lines – grid cells

11. Departure volume calculation Red lines – Lagrangian (departure) latitudes; Red stars – corners of the departure volume; Green pentagons – points where Lagrangian latitudes cross longitudes of the arrival grid (points of the intermediate grid); green dotted lines – midpoint values for latitudes of the intermediate grid defining north and south boundaries of the intermediate grid cells (grey rectangles). Orange lines – approximation for west (east) boundaries of departure volume

12. CCS on the reduced grid (1) MM - mass of the advected quantity bounded in the cell of arrival (computational) grid. M * - mass bounded in the cell of departure grid. Direct conservative cascade remapping of the mass from the reduced grid to any kind of departure grid is impossible due to the absence of well defined grid longitudes matching poles of the sphere in the reduced grid. So we implemented slightly different scheme: MnMn M n * = M n+1 Conservative cascade remapping M n+1 * Conservative cascade remapping MnMn Conservative remapping M n reg M n * = M n+1 Conservative cascade remapping TThe reduced grid is used as computational and arrival grid. M n reg - masses of the advected quantity bounded in the cells of regular grid. The structure of the departure grid corresponding to the reduced arrival grid is suitable for CCS since such departure grid forms well-defined intermediate cells (see Nair et al. for details)

13. CCS on the reduced grid (2) We have chosen regular grid such that latitude grid lines of regular and reduced grid coincide and the number of grid points in latitude line of regular grid is equal to the maximum number of grid points in the latitude lines of the reduced grid. 1D remapping technique we use for remapping between reduced and regular grids is the same as the one used in 2D conservative cascade remapping The first 1D remapping in the CCS is in latitudinal direction. The application of conservative scheme using first remapping in longitudinal direction (ex. SLICE by Zerroukat et al.) on the reduced grid is much more complicated.

14. Mass-conservative Helmholtz problem solver Semi-implicit time discretization of the SW equations leads to the Helmholtz problem to be solved at each time step: Φ – height, Φ ref – reference height, F – some known function, D – divergence of the wind field, A – some known vector, μ is a constant. If S i, j – the volume of i, j -th cell of computational grid then the mass conservation is expressed by The mass-conservation can be achieved if discrete Laplacian and divergence operators satisfy These operators are constructed with a finite-volume approach (in Zerroukat et al 2009, FDs are used)

15. Results of standard tests - preliminaries Tests were carried out using the regular grids of 1.5, 1, 0.75, 0.5 degrees resolution in latitude and longitude and the reduced grid of 1.5 and 0.75 degrees resolution in latitude and the same maximum resolution in longitude. Test cases ## 1, 7 were carried out only on the regular and reduced grids of 1.5 degree resolution. Reduced grid for the tests was constructed by R. Fadeev (Fadeev R.Yu., Construction of a reduced latitude-longitude grid for a global numerical weather prediction problem, Russian Meteorology and Hydrology, 2006, N 9). Reduction rate was 10%. For comparison, the same tests were carried out with the original non- conservative SW model. In the tables, the abbreviation SL denotes the original non-conservative model and CM denotes its mass conservative modification Resolution1.5 0 1.0 0 0.75 0 0.5 0 Timestep(sec)270018001350900

16. Test case 1 - advection over the pole Resolution in latitude and longitude (maximum resolution for reduced grid) - 1,5 0. Meridional Courant number 0.5 Resolutionl 1 x10 -2 l 2 x10 -2 l ∞ x10 -2 MAX x 10 -2 CM regular grid1.5 0 1.971.34 -0.4 CM reduced grid1.5 0 1.971.34 -0.4 SL regular grid1.5 0 6.033.783.10-1.8 Height error measures after 1 revolution The initial distribution (cosine bell) (A). Error fields after one revolution – CM on regular grid (B), CM on reduced grid (C), SL on regular grid (D) CABDC

17. Test case 2 – Global steady state geostrophic flow Angle α = π/4. Resolution1.5 0 1.0 0.75 0 Gridreducedregular reducedregular CM0.50E-040.18E-040.86E-050.11E-040.45E-05 SL0.29E-040.22E-040.90E-050.67E-050.50E-05 l2 height error after 5 days of modeling l2 height error for CM on the regular grid with resolution 1.5 0 (red line), 1.0 0 (green line),.75 0 (blue line)

18. Test case 6 – Rossby-Haurwitz wave #4 l2 height error after 14 days of modeling Height field after 14 days of modeling with CM (regular grid 1.5 0 of resolution). Reference solution (A), CM error (B), SL error (C) (all on regular grid). Resolution1.5 0 1.0 0.75 0.5 0 Gridreducedregular reducedregular CM0.47E-04 0.30E-040.23E-040.22E-040.18E-04 SL0.48E-04 0.30E-040.23E-04 0.18E-04 ABC

19. Test cases 7 – Real data Tests were carried out with the fixed resolution of 1.5 0. l2 height error after 5 days of modeling Test case7a 7b 7c Gridreducedregularreducedregularreducedregular CM0.17E-02 0.21E-02 0.22E-02 SL0.17E-02 0.22E-02 0.23E-02 ABC

20. Galewsky case 4th order diffusion was applied to divergence field only. CM solution on the regular grid with the resolution of 0.325 0 is a reference. Vorticity field after 144h. CM (regular grid): 1. 0 – A,.75 0 – B,.5 0 – C,.325 0 – D. Panel E - vorticity errors with respect to the reference solution AB C AB C D E

21. New solver for U-V reconstruction (1) Old solver: solution of 2 Poisson equations, + differentiation using Helmholtz relations In a parallel environment, requires given Fourier coefficient at all latitudinal points. So the need for data transposition that limits scalability.

22. New solver for U-V reconstruction (2) New solver: direct solution of the system => Possibility to apply a parallel solver. No need for data transposition.

23. New solver for U-V reconstruction (3),,.

24. Normalized L2-error for U in 3D steady-state test 1 by Jablonowski et al: old solver (left), new solver (right) New solver also can provide better parallelization using parallel dichotomy algorithm (Terekhov, Par. Computing 2010)

25. Conclusions Mass-conservative shallow-water version of SL-AV model was created. The error measures in standard SW tests are the same for conservative and non-conservative versions, except for test #1 where conservative scheme gives smaller errors, and test #2 where conservative scheme gives larger errors. The impact of reduced grid on errors is negligible except for test #2. The new solver implemented for U-V reconstruction is cheaper, more accurate and more scalable. Further work – to implement mass-conservative version of the 3D model. There is also ongoing work on NH dynamical core development

26. Thank you for attention!

27. Mass-conservative Helmholtz problem solver (2) To obtain discrete divergence operator, the cell-averaged divergence is represented as a flux of vector field through the cell-boundaries. If C i,j is an arbitrary non-polar grid cell, and (u,v) is an vector field then: I east, I west, I north, I south designate flux of (u,v) through one of the cell boundaries. Since cell boundaries are parallel to the coordinate axes, flux operators I are defined by straightforward expressions. If C 0 is the south polar cap, and C i,1, i=1,M are the cells around, then polar cap averaged divergence is defined as follows Flux operators and so cell-averaged divergence operator are discretized with fourth-order accuracy. Laplacian is discretized as cell-averaged divergence of gradient with second order. Divergence and Laplacian discretized in this way satisfy the last equality from the previous slide and produce conservative Helmholtz problem solver implying tridiagonal matrix inversion. Another way to achieve mass-conservation is the use of conservative finite- differences for discretization of Helmholtz equation (as in Zerroukat et al. 2009)

28. Non-conservative model: relative total mass errors Regular grid with the resolution 1.5 0 – red lines, 1 0 – green lines, 0.75 0 – blue lines; tests #1 (A), #2 (B), #6 (C), #7a (D) AB C D B C A D