1. Massively Parallel Magnetohydrodynamics on the Cray XT3 Joshua Breslau and Jin Chen Princeton Plasma Physics Laboratory Cray XT3 Technical Workshop Nashville, TN February 28, 2007

2. Motivation: Modeling Magnetic Confinement Fusion Experiments NSTX (Spherical Torus) NCSX (Compact Stellarator) ITER (Advanced Tokamak)

3. Characteristics of Magnetic Confinement Fusion Experiments Multispecies hydrogen plasma at ~10 8 °C (T e and T i may differ): low collisionality, high electrical conductivity. Toroidal topology with complex boundary geometry. Strong toroidal magnetic field giving highly anisotropic transport, low . Rotational transform gives nested flux surfaces. Spatial scales range from electron skin depth, ~10 -4 m to major radius, ~6 m. Time scales range from Alfvén wave transit time (~  s) to discharge time, ~100 s. Susceptible to microinstabilities leading to loss of energy confinement; and macroinstabilities leading to large-scale rearrangment of plasma and possible disruption.

4. Extended MHD Equations

5. The M3D Code Physics models include ideal and resistive MHD; two-fluid; or hybrid with kinetic ions. Field and velocity variables are expressed in terms of potentials, keeping B divergence-free and separating compressible and incompressible components of flow. Uses linear, 2 nd, or 3 rd -order finite elements in-plane on an unstructured triangular mesh. Uses 4 th -order finite differences between planes or pseudo-spectral derivatives. Partially implicit treatment allows efficient advance over dissipative time scales but requires small time steps relative to  A. Linear and nonlinear modes of operation are available. The PETSc library is used for parallelization and linear solves with Krylov methods. M3D (multi-level 3D) is a 3D nonlinear extended MHD code in toroidal geometry maintained by a multi-institutional collaboration, designed for the study of macroscopic instabilities in tokamaks and stellarators.

6. M3D Mesh Single planeFull torus Radial zones align with flux surfaces. 2n+1 planes needed to resolve toroidal mode #n.

7. Domain Decomposition Poloidal (cross-section view) Toroidal (overhead view) or D = 1 F = 5 D = 3 F = 3 B = 16 Linear solves independent on each processorLinear solves parallel over processors

8. Porting M3D to the XT3 Previously run on Cray T3E, IBM SP, SGI Origin 2000. Few modifications to source code were necessary for the new platform. Installation of HYPRE preconditioner in PETSc library made possible faster inversion of symmetric form of linear operators using CG. Reducing interprocessor communication was key to improving scaling.

9. 1D Weak Scaling, Single Core (SN) 560 radial zones (626,081 vertices/plane), 16 poloidal processors, 4 planes/processor, 4-320 toroidal processors poloidal domain decomposition

10. 1D Weak Scaling, Dual Core (VN) 398 radial zones (316,013 vertices/plane), 16 poloidal processors, 4 planes/processor, 4-640 toroidal processors

11. Smallest run has 64 planes, 160,000 vertices/plane on 16 toroidal x 4 poloidal processors. Successive runs increase number of poloidal processors by 4, number of toroidal processors by 12, while maintaining 4 planes, 40,000 vertices/processor. 3D Weak Scaling, Single Core

12. 3D Strong Scaling, Single Core All runs have 32 planes, 474,151 vertices/plane. Smallest has 8 toroidal x 12 poloidal processors. Successive runs increase number of poloidal processors by 6, double number of toroidal processors.

13. 3D Strong Scaling, Single Core All runs have 128 planes, 474,151 vertices/plane. Smallest has 32 toroidal x 12 poloidal processors. Successive runs increase number of poloidal processors by 6, double number of toroidal processors.

14. 3D Strong Scaling, Single Core All runs have 208 planes, 474,151 vertices/plane. Smallest has 52 toroidal x 12 poloidal processors. Successive runs increase number of poloidal processors by 6, double number of toroidal processors.

15. Sample Application: CDX Sawteeth High temperature core X-point (site of reconnection) Low temperature m=1, n=1 island q=1 surface (inversion radius) Small laboratory tokamak Oscillations in X-ray signal during discharge consistent with sudden outward shift of hot plasma Objective: predict effect and conditions for onset of instability.

16. Initialization Equilibrium taken from a transport-timescale code.  ~ 3.3% q 0  0.922 Sawtooth instability is predicted when q 0 is sufficiently below 1. toroidal current density Linear n=1 eigenmode:  A  6  10 -4 Perturbed temperaturePerturbed current densityVelocity stream function

17. Nonlinear Results 24 planes, 79 radial grids 24 toroidal x 6 poloidal processors 221,856 vertices on 144 Jaguar CPUs (VN mode) Kinetic energy, by toroidal mode number Poincaré Sections 13,920 CPU hours (96:40 wallclock hours)

18. Conclusions The XT3 has been a productive environment for tokamak simulations with M3D. Improved scaling can be expected with the faster interconnects on the XT4. Scaling to thousands of processors has been demonstrated, but may be impractical for real applications while the code remains explicit.