Parallel Solution of Navier Stokes Equations Xing Cai Dept. of Informatics University of Oslo

Outline of the Talk Two parallelization strategiesTwo parallelization strategies –based on domain decomposition –at the linear algebra level Parallelization of Navier-StokesParallelization of Navier-Stokes Numerical experimentsNumerical experiments

Diffpack O-O software environment for scientific computationO-O software environment for scientific computation Rich collection of PDE solution components - portable, flexible, extensibleRich collection of PDE solution components - portable, flexible, extensible H.P.Langtangen: Computational Partial Differential Equations, Springer 1999H.P.Langtangen: Computational Partial Differential Equations, Springer 1999

The Question Starting point: sequential PDE solvers How to do the parallelization? Resulting parallel solvers should have 4 good parallel efficiency 4 good overall numerical performance We need 4 a good parallelization strategy 4 a good and simple implementation of the strategy

Domain Decomposition Solution of the original large problem through iteratively solving many smaller subproblemsSolution of the original large problem through iteratively solving many smaller subproblems Can be used as solution method or preconditionerCan be used as solution method or preconditioner Flexibility -- localized treatment of irregular geometries, singularities etcFlexibility -- localized treatment of irregular geometries, singularities etc Very efficient numerical methods -- even on sequential computersVery efficient numerical methods -- even on sequential computers Suitable for coarse grained parallelizationSuitable for coarse grained parallelization

Additive Schwarz Method Subproblems can be solved in parallelSubproblems can be solved in parallel Subproblems are of the same form as the original large problem, with possibly different boundary conditions on artificial boundariesSubproblems are of the same form as the original large problem, with possibly different boundary conditions on artificial boundaries

Convergence of the Solution Single-phasegroundwaterflow

Observations DD is a good parallelization strategyDD is a good parallelization strategy The approach is not PDE-specificThe approach is not PDE-specific A program for the original global problem can be reused (modulo B.C.) for each subdomainA program for the original global problem can be reused (modulo B.C.) for each subdomain Must communicate overlapping point valuesMust communicate overlapping point values No need for global dataNo need for global data Explicit temporal schemes are a special case where no iteration is needed (“exact DD”)Explicit temporal schemes are a special case where no iteration is needed (“exact DD”)

Goals for the Implementation Reuse sequential solver as subdomain solverReuse sequential solver as subdomain solver Add DD management and communication as separate modulesAdd DD management and communication as separate modules Collect common operations in generic library modulesCollect common operations in generic library modules Flexibility and portabilityFlexibility and portability Simplified parallelization process for the end-userSimplified parallelization process for the end-user

Generic Programming Framework

Generic Subdomain Simulators SubdomainSimulatorSubdomainSimulator –abstract interface to all subdomain simulators, as seen by the Administrator SubdomainFEMSolverSubdomainFEMSolver –Special case of SubdomainSimulator for finite element-based simulators These are generic classes, not restricted to specific application areasThese are generic classes, not restricted to specific application areas SubdomainFEMSolver SubdomainSimulator

Making the Simulator Parallel class SimulatorP : public SubdomainFEMSolver public Simulator public Simulator{ // … just a small amount of code // … just a small amount of code virtual void createLocalMatrix () virtual void createLocalMatrix () { Simulator::makeSystem (); } { Simulator::makeSystem (); }}; SubdomainSimulator SubdomainFEMSolver Administrator SimulatorP Simulator

Summary So Far A generic approachA generic approach Works if the DD algorithm worksWorks if the DD algorithm works Make use of class hierarchiesMake use of class hierarchies The new parallel-specific code, SimulatorP, is very small and simple to writeThe new parallel-specific code, SimulatorP, is very small and simple to write

Application  Single-phase groundwater flow  DD as the global solution method  Subdomain solvers use CG+FFT  Fixed number of subdomains M =32 (independent of P )  Straightforward parallelization of an existing simulator P: number of processors

Linear-algebra-level Approach Parallelize matrix/vector operationsParallelize matrix/vector operations –inner-product of two vectors –matrix-vector product –preconditioning - block contribution from subgrids Easy to useEasy to use –access to all Diffpack v3.0 iterative methods, preconditioners and convergence monitors –“hidden” parallelization –need only to add a few lines of new code –arbitrary choice of number of procs at run-time –less flexibility than DD

Straightforward Parallelization Develop a sequential simulator, without paying attention to parallelismDevelop a sequential simulator, without paying attention to parallelism Follow the Diffpack coding standardsFollow the Diffpack coding standards Need Diffpack add-on libraries for parallel computingNeed Diffpack add-on libraries for parallel computing Add a few new statements for transformation to a parallel simulatorAdd a few new statements for transformation to a parallel simulator

Library Tool class GridPartAdmclass GridPartAdm –Generate overlapping or non-overlapping subgrids –Prepare communication patterns –Update global values –matvec, innerProd, norm

A Simple Coding Example GridPartAdm* adm; // access to parallelizaion functionality LinEqAdm* lineq; // administrator for linear system & solver //... #ifdef PARALLEL_CODE adm->scan (menu); adm->prepareSubgrids (); adm->prepareCommunication (); lineq->attachCommAdm (*adm); #endif //... lineq->solve (); set subdomain list = DEFAULT set global grid = grid1.file set partition-algorithm = METIS set number of overlaps = 0

Single-phase Groundwater Flow Highly unstructured grid Highly unstructured grid Discontinuity in the coefficient K (0.1 & 1) Discontinuity in the coefficient K (0.1 & 1)

Measurements 130,561 degrees of freedom 130,561 degrees of freedom Overlapping subgrids Overlapping subgrids Global BiCGStab using (block) ILU prec. Global BiCGStab using (block) ILU prec.

Test Case: Vortex-Shedding

Simulation Snapshots Pressure

Animated Pressure Field

Some CPU-Measurements The pressure equation is solved by the CG method

Combined Approach Use a CG-like method as basic solverUse a CG-like method as basic solver (i.e. use a parallelized Diffpack linear solver) Use DD as preconditionerUse DD as preconditioner (i.e. SimulatorP is invoked as a preconditioner solve) Combine with coarse grid correctionCombine with coarse grid correction CG-like method + DD prec. is normally faster than DD as a basic solverCG-like method + DD prec. is normally faster than DD as a basic solver

Two-phase Porous Media Flow PEQ: SEQ: BiCGStab + DD prec. for global pressure eq. Multigrid V-cycle in subdomain solves

Two-phase Porous Media Flow History of saturation for water and oil

Nonlinear Water Waves Fully nonlinear 3D water waves Primary unknowns: Parallelization based on an existing sequential Diffpack simulator

Nonlinear Water Waves CG + DD prec. for global solverCG + DD prec. for global solver Multigrid V-cycle as subdomain solverMultigrid V-cycle as subdomain solver Fixed number of subdomains M =16 (independent of P )Fixed number of subdomains M =16 (independent of P ) Subgrids from partition of a global 41x41x41 gridSubgrids from partition of a global 41x41x41 grid

Nonlinear Water Waves 3D Poisson equation in water wave simulation

Summary Goal: provide software and programming rules for easy parallelization of sequential simulatorsGoal: provide software and programming rules for easy parallelization of sequential simulators Two parallelization strategies:Two parallelization strategies: –domain decomposition: very flexible, compact visible code/algorithm very flexible, compact visible code/algorithm –parallelization at the linear algebra level: “automatic” hidden parallelization “automatic” hidden parallelization Performance: satisfactory speed-upPerformance: satisfactory speed-up