Massively Parallel Magnetohydrodynamics on the Cray XT3 Joshua Breslau and Jin Chen Princeton Plasma Physics Laboratory Cray XT3 Technical Workshop Nashville,

Slides:

Advertisements

Similar presentations

EXTENDED MHD MODELING: BACKGROUND, STATUS AND VISION

Advertisements

Dynamo Effects in Laboratory Plasmas S.C. Prager University of Wisconsin October, 2003.

Progress and Plans on Magnetic Reconnection for CMSO For NSF Site-Visit for CMSO May1-2, Experimental progress [M. Yamada] -Findings on two-fluid.

EXTENDED MHD SIMULATIONS: VISION AND STATUS D. D. Schnack and the NIMROD and M3D Teams Center for Extended Magnetohydrodynamic Modeling PSACI/SciDAC.

November 3-5, 2003Feedback Workshop, Austin NORMAL MODE APPROACH TO MODELING OF FEEDBACK STABILIZATION OF THE RESISTIVE WALL MODE By M.S. Chu(GA), M.S.

Particle acceleration in a turbulent electric field produced by 3D reconnection Marco Onofri University of Thessaloniki.

Seminario UT FUSIONE Aula Brunelli, Centro Ricerche Frascati 8 Febbraio 2010 Ideal MHD Stability Boundaries of the PROTO-SPHERA Configuration F. Alladio,

Halo Current and Resistive Wall Simulations of ITER H.R. Strauss 1, Linjin Zheng 2, M. Kotschenreuther 2, W.Park 3, S. Jardin 3, J. Breslau 3, A.Pletzer.

Algorithm Development for the Full Two-Fluid Plasma System

A Kinetic-Fluid Model for Studying Thermal and Fast Particle Kinetic Effects on MHD Instabilities C. Z. Cheng, N. Gorelenkov and E. Belova Princeton Plasma.

Cyclic MHD Instabilities Hartmut Zohm MPI für Plasmaphysik, EURATOM Association Seminar talk at the ‚Advanced Course‘ of EU PhD Network, Garching, September.

William Daughton Plasma Physics Group, X-1 Los Alamos National Laboratory Presented at: Second Workshop on Thin Current Sheets University of Maryland April.

Physics of fusion power Lecture 6: Conserved quantities / Mirror device / tokamak.

INTRODUCTION OF WAVE-PARTICLE RESONANCE IN TOKAMAKS J.Q. Dong Southwestern Institute of Physics Chengdu, China International School on Plasma Turbulence.

Physics of fusion power

Physics of fusion power Lecture 8: Conserved quantities / mirror / tokamak.

Physics of fusion power

GTC Status: Physics Capabilities & Recent Applications Y. Xiao for GTC team UC Irvine.

Magnetic Reconnection in Multi-Fluid Plasmas Michael Shay – Univ. of Maryland.

Physics of Fusion power Lecture 7: Stellarator / Tokamak.

Non-disruptive MHD Dynamics in Inward-shifted LHD Configurations 1.Introduction 2.RMHD simulation 3.DNS of full 3D MHD 4. Summary MIURA, H., ICHIGUCHI,

Computer simulations of fast frequency sweeping mode in JT-60U and fishbone instability Y. Todo (NIFS) Y. Shiozaki (Graduate Univ. Advanced Studies) K.

Iterative and direct linear solvers in fully implicit magnetic reconnection simulations with inexact Newton methods Xuefei (Rebecca) Yuan 1, Xiaoye S.

Presented by XGC: Gyrokinetic Particle Simulation of Edge Plasma CPES Team Physics and Applied Math Computational Science.

SIMULATION OF A HIGH-  DISRUPTION IN DIII-D SHOT #87009 S. E. Kruger and D. D. Schnack Science Applications International Corp. San Diego, CA USA.

6 th Japan-Korea Workshop on Theory and Simulation of Magnetic Fusion Plasmas Hyunsun Han, G. Park, Sumin Yi, and J.Y. Kim 3D MHD SIMULATIONS.

Kinetic Effects on the Linear and Nonlinear Stability Properties of Field- Reversed Configurations E. V. Belova PPPL 2003 APS DPP Meeting, October 2003.

Overview of MHD and extended MHD simulations of fusion plasmas Guo-Yong Fu Princeton Plasma Physics Laboratory Princeton, New Jersey, USA Workshop on ITER.

Hybrid Simulations of Energetic Particle-driven Instabilities in Toroidal Plasmas Guo-Yong Fu In collaboration with J. Breslau, J. Chen, E. Fredrickson,

Introduction to the Particle In Cell Scheme for Gyrokinetic Plasma Simulation in Tokamak a Korea National Fusion Research Institute b Courant Institute,

Challenging problems in kinetic simulation of turbulence and transport in tokamaks Yang Chen Center for Integrated Plasma Studies University of Colorado.

Multiscale issues in modeling magnetic reconnection J. F. Drake University of Maryland IPAM Meeting on Multiscale Problems in Fusion Plasmas January 10,

Numerical Simulation on Flow Generated Resistive Wall Mode Shaoyan Cui (1,2), Xiaogang Wang (1), Yue Liu (1), Bo Yu (2) 1.State Key Laboratory of Materials.

ParCFD Parallel computation of pollutant dispersion in industrial sites Julien Montagnier Marc Buffat David Guibert.

Nonlinear Extended MHD Simulation Using High-Order Finite Elements C. Sovinec, C. Kim, H. Tian, Univ. of WI-Madison D. Schnack, A. Pankin, Science Apps.

Electron behaviour in three-dimensional collisionless magnetic reconnection A. Perona 1, D. Borgogno 2, D. Grasso 2,3 1 CFSA, Department of Physics, University.

DIII-D SHOT #87009 Observes a Plasma Disruption During Neutral Beam Heating At High Plasma Beta Callen et.al, Phys. Plasmas 6, 2963 (1999) Rapid loss of.

Nonlinear interactions between micro-turbulence and macro-scale MHD A. Ishizawa, N. Nakajima, M. Okamoto, J. Ramos* National Institute for Fusion Science.

MHD Dynamo Simulation by GeoFEM Hiroaki Matsui Research Organization for Informatuion Science & Technology(RIST), JAPAN 3rd ACES Workshop May, 5, 2002.

Electron inertial effects & particle acceleration at magnetic X-points Presented by K G McClements 1 Other contributors: A Thyagaraja 1, B Hamilton 2,

M. Onofri, F. Malara, P. Veltri Compressible magnetohydrodynamics simulations of the RFP with anisotropic thermal conductivity Dipartimento di Fisica,

Hybrid MHD-Gyrokinetic Simulations for Fusion Reseach G. Vlad, S. Briguglio, G. Fogaccia Associazione EURATOM-ENEA, Frascati, (Rome) Italy Introduction.

Team Report on integration of FSAM to SWMF and on FSAM simulations of convective dynamo and emerging flux in the solar convective envelope Yuhong Fan and.

STUDIES OF NONLINEAR RESISTIVE AND EXTENDED MHD IN ADVANCED TOKAMAKS USING THE NIMROD CODE D. D. Schnack*, T. A. Gianakon**, S. E. Kruger*, and A. Tarditi*

EXTENSIONS OF NEOCLASSICAL ROTATION THEORY & COMPARISON WITH EXPERIMENT W.M. Stacey 1 & C. Bae, Georgia Tech Wayne Solomon, Princeton TTF2013, Santa Rosa,

1 A Proposal for a SWIM Slow-MHD 3D Coupled Calculation of the Sawtooth Cycle in the Presence of Energetic Particles Josh Breslau Guo-Yong Fu S. C. Jardin.

Parallel coupling: problems arising in the context of magnetic fusion John R. Cary Professor, University of Colorado CEO, Tech-X Corporation.

NIMROD Simulations of a DIII-D Plasma Disruption

QAS Design of the DEMO Reactor

Simulations of NBI-driven Global Alfven Eigenmodes in NSTX E. V. Belova, N. N. Gorelenkov, C. Z. Cheng (PPPL) NSTX Results Forum, PPPL July 2006 Motivation:

The Joint Meeting of 4th IAEA Technical Meeting on Spherical Tori and 14th International Workshop on Spherical TorusFrascati, 7 to 10 October Ideal.

Presented by High Performance Computing in Magnetic Fusion Energy Research Donald B. Batchelor RF Theory Plasma Theory Group Fusion Energy Division.

21st IAEA Fusion Energy Conf. Chengdu, China, Oct.16-21, /17 Gyrokinetic Theory and Simulation of Zonal Flows and Turbulence in Helical Systems T.-H.

Center for Extended MHD Modeling (PI: S. Jardin, PPPL) –Two extensively developed fully 3-D nonlinear MHD codes, NIMROD and M3D formed the basis for further.

Simulations of Energetic Particle Modes In Spherical Torus G.Y. Fu, J. Breslau, J. Chen, E. Fredrickson, S. Jardin, W. Park Princeton Plasma Physics Laboratory.

Nonlinear Simulations of Energetic Particle-driven Modes in Tokamaks Guoyong Fu Princeton Plasma Physics Laboratory Princeton, NJ, USA In collaboration.

Yang Henglei( 杨恒磊 ),Wang Xiaogang( 王晓钢 ) State Key Laboratory of Materials Modification by Laser, Ion and Electron Beams; The Department of Physics;

Simulation of Turbulence in FTU M. Romanelli, M De Benedetti, A Thyagaraja* *UKAEA, Culham Sciance Centre, UK Associazione.

Presented by Yuji NAKAMURA at US-Japan JIFT Workshop “Theory-Based Modeling and Integrated Simulation of Burning Plasmas” and 21COE Workshop “Plasma Theory”

S j l ij l ik l il i j k l C C i C j C k C l S i S k S l m ij m ik m il s i Roberto Lionello and Dalton D. Schnack Center for Energy and Space Science.

Resistive Modes in CDX-U J. Breslau, W. Park. S. Jardin, R. Kaita – PPPL D. Schnack, S. Kruger – SAIC APS-DPP Annual Meeting Albuquerque, NM October 30,

Plan V. Rozhansky, E. Kaveeva St.Petersburg State Polytechnical University, , Polytechnicheskaya 29, St.Petersburg, Russia Poloidal and Toroidal.

Energetic ion excited long-lasting “sword” modes in tokamak plasmas with low magnetic shear Speaker:RuiBin Zhang Advisor:Xiaogang Wang School of Physics,

NIMROD Simulations of a DIII-D Plasma Disruption S. Kruger, D. Schnack (SAIC) April 27, 2004 Sherwood Fusion Theory Meeting, Missoula, MT.

Reconnection Process in Sawtooth Crash in the Core of Tokamak Plasmas Hyeon K. Park Ulsan National Institute of Science and Technology, Ulsan, Korea National.

Unstructured Meshing Tools for Fusion Plasma Simulations

Adaptive Grid Generation for Magnetically Confined Plasmas

Influence of energetic ions on neoclassical tearing modes

Non linear evolution of 3D magnetic reconnection in slab geometry

Stellarator Program Update: Status of NCSX & QPS

Presentation transcript:

Massively Parallel Magnetohydrodynamics on the Cray XT3 Joshua Breslau and Jin Chen Princeton Plasma Physics Laboratory Cray XT3 Technical Workshop Nashville, TN February 28, 2007

Motivation: Modeling Magnetic Confinement Fusion Experiments NSTX (Spherical Torus) NCSX (Compact Stellarator) ITER (Advanced Tokamak)

Characteristics of Magnetic Confinement Fusion Experiments Multispecies hydrogen plasma at ~10 8 °C (T e and T i may differ): low collisionality, high electrical conductivity. Toroidal topology with complex boundary geometry. Strong toroidal magnetic field giving highly anisotropic transport, low . Rotational transform gives nested flux surfaces. Spatial scales range from electron skin depth, ~10 -4 m to major radius, ~6 m. Time scales range from Alfvén wave transit time (~  s) to discharge time, ~100 s. Susceptible to microinstabilities leading to loss of energy confinement; and macroinstabilities leading to large-scale rearrangment of plasma and possible disruption.

Extended MHD Equations

The M3D Code Physics models include ideal and resistive MHD; two-fluid; or hybrid with kinetic ions. Field and velocity variables are expressed in terms of potentials, keeping B divergence-free and separating compressible and incompressible components of flow. Uses linear, 2 nd, or 3 rd -order finite elements in-plane on an unstructured triangular mesh. Uses 4 th -order finite differences between planes or pseudo-spectral derivatives. Partially implicit treatment allows efficient advance over dissipative time scales but requires small time steps relative to  A. Linear and nonlinear modes of operation are available. The PETSc library is used for parallelization and linear solves with Krylov methods. M3D (multi-level 3D) is a 3D nonlinear extended MHD code in toroidal geometry maintained by a multi-institutional collaboration, designed for the study of macroscopic instabilities in tokamaks and stellarators.

M3D Mesh Single planeFull torus Radial zones align with flux surfaces. 2n+1 planes needed to resolve toroidal mode #n.

Domain Decomposition Poloidal (cross-section view) Toroidal (overhead view) or D = 1 F = 5 D = 3 F = 3 B = 16 Linear solves independent on each processorLinear solves parallel over processors

Porting M3D to the XT3 Previously run on Cray T3E, IBM SP, SGI Origin Few modifications to source code were necessary for the new platform. Installation of HYPRE preconditioner in PETSc library made possible faster inversion of symmetric form of linear operators using CG. Reducing interprocessor communication was key to improving scaling.

1D Weak Scaling, Single Core (SN) 560 radial zones (626,081 vertices/plane), 16 poloidal processors, 4 planes/processor, toroidal processors poloidal domain decomposition

1D Weak Scaling, Dual Core (VN) 398 radial zones (316,013 vertices/plane), 16 poloidal processors, 4 planes/processor, toroidal processors

Smallest run has 64 planes, 160,000 vertices/plane on 16 toroidal x 4 poloidal processors. Successive runs increase number of poloidal processors by 4, number of toroidal processors by 12, while maintaining 4 planes, 40,000 vertices/processor. 3D Weak Scaling, Single Core

3D Strong Scaling, Single Core All runs have 32 planes, 474,151 vertices/plane. Smallest has 8 toroidal x 12 poloidal processors. Successive runs increase number of poloidal processors by 6, double number of toroidal processors.

3D Strong Scaling, Single Core All runs have 128 planes, 474,151 vertices/plane. Smallest has 32 toroidal x 12 poloidal processors. Successive runs increase number of poloidal processors by 6, double number of toroidal processors.

3D Strong Scaling, Single Core All runs have 208 planes, 474,151 vertices/plane. Smallest has 52 toroidal x 12 poloidal processors. Successive runs increase number of poloidal processors by 6, double number of toroidal processors.

Sample Application: CDX Sawteeth High temperature core X-point (site of reconnection) Low temperature m=1, n=1 island q=1 surface (inversion radius) Small laboratory tokamak Oscillations in X-ray signal during discharge consistent with sudden outward shift of hot plasma Objective: predict effect and conditions for onset of instability.

Initialization Equilibrium taken from a transport-timescale code.  ~ 3.3% q 0  Sawtooth instability is predicted when q 0 is sufficiently below 1. toroidal current density Linear n=1 eigenmode:  A  6  Perturbed temperaturePerturbed current densityVelocity stream function

Nonlinear Results 24 planes, 79 radial grids 24 toroidal x 6 poloidal processors 221,856 vertices on 144 Jaguar CPUs (VN mode) Kinetic energy, by toroidal mode number Poincaré Sections 13,920 CPU hours (96:40 wallclock hours)

Conclusions The XT3 has been a productive environment for tokamak simulations with M3D. Improved scaling can be expected with the faster interconnects on the XT4. Scaling to thousands of processors has been demonstrated, but may be impractical for real applications while the code remains explicit.