1. High Performance Computing Seminar II Parallel mesh partitioning with ParMETIS Parallel iterative solvers with Hypre M.Sc. Caroline Mendonça Costa

2. Caroline Mendonça Costa ParMETIS and Hypre Page 2 Outline ParMETIS ● Review of Multilevel K- way partitioning ● Parallel K-way partitioning ● Recursive bisection ● Geometric partitioning ● Work in progress PETSc and Hypre ● Solution of PDE's ● CARP ● Large linear systems ● PCG (PETSc) ● Preconditioning ● AMG ● BoomerAMG (Hypre) ● Performance comparison

3. Caroline Mendonça Costa ParMETIS and Hypre Page 3 Parallel mesh partitioning with ParMETIS

4. Caroline Mendonça Costa ParMETIS and Hypre Page 4 Overview of the package ● Parallel version of the Metis mesh partitioner ● Developed by the Karypis lab at the Digital Technology Center of the University of Minnesota ● Can be downloaded at ● http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download ● What it can do... ● Mesh/graph partitioning ● Graph repartitioning ● Partition refinement ● Matrix reordering

5. Caroline Mendonça Costa ParMETIS and Hypre Page 5 K-way graph partitioning problem

6. Caroline Mendonça Costa ParMETIS and Hypre Page 6 Multilevel K-way partitioning Nodes are collapsed together Matching Multilevel Recursive Bisection Initial K-way partitioning Partitioning is projected onto the coarser graphs Partitioning refinement

7. Caroline Mendonça Costa ParMETIS and Hypre Page 7 Parallel multilevel K-way partitioning Graph coloring algorithm ● Luby's algorithm Parallel coloring ● Distributed memory ● Communication setup ● Exchange of random numbers ● Single augmentation step ● Drawbacks ● Independent set is not maximal ● Increase number of colors ● Not all nodes are colored ● Advantages ● Reduces the overall time ● Most nodes participate

8. Caroline Mendonça Costa ParMETIS and Hypre Page 8 Parallel multilevel K-way partitioning Coarsening phase ● Parallel matching algorithm ● Uses the coloring of the graph to structure the computations ● During the c th iteration, the vertices of color c select one of their unmatched neighbors using HEM heuristics ● After each iteration, the vertices synchronize ● Distributed memory ● The write and read operations are gathered together and sent in a single message ● During the read operation the processors also determine if they will store the collapsed vertex

9. Caroline Mendonça Costa ParMETIS and Hypre Page 9 Parallel multilevel K-way partitioning Uncoarsening/refinement phase Uncoarsening ● Each processor contains one subgraph ● Each subgraph is expanded independently Refinement ● Uses the coloring to structure the vertex swaps ● During the cth iteration all vertices of color c are considered for movement ● The subset that leads to a reduction in the EC is “moved” ● Distributed memory ● The vertices do not move immediately: the partition number is updated ● By the end of the algorithm the vertices are moved to the corresponding processor

10. Caroline Mendonça Costa ParMETIS and Hypre Page 10 Parallel multilevel K-way partitioning K-way partitioning ● Nested dissection ● Parallelization? Geometric partitioning ● Space-filling curves ● Parallelization?

11. Caroline Mendonça Costa ParMETIS and Hypre Page 11 Work in progress Metis x ParMetis ● Metis: stand-alone software and library ● ParMetis: only distributed as a library Using ParMetis to partitioning meshes ● A few steps... ● Obtain graph from mesh ● Include geometric information – Not working ● Use the graph partitioning routines ● K-way partitioning ● Geometric K-way partitioning ● Challenges ● Structure of the code is not well described in the manual or in the code

12. Caroline Mendonça Costa ParMETIS and Hypre Page 12 Parallel Iterative Solvers with Hypre

13. Caroline Mendonça Costa ParMETIS and Hypre Page 13 Solution of PDE systems Bidomain model Elliptic system Nonlinear ODE system Parabolic PDE Operator Splitting FE M Bottleneck of computation Large linear system

14. Caroline Mendonça Costa ParMETIS and Hypre Page 14 Solution of PDE systems CARP – Cardiac Arrhythmia Research Package Contact Dr. Gernot Plank at the Institute of Biophysics of the MedUniGraz

15. Caroline Mendonça Costa ParMETIS and Hypre Page 15 Large Linear Systems sM = r Preconditioned Conjugate Gradient Apply different iterative solver to the residual (defect) system AM G

16. Caroline Mendonça Costa ParMETIS and Hypre Page 16 Preconditioning Algebraic Multigrid ● Setup phase ● Assemble ● Coarse levels ● Restriction ● Prolongation Restriction Interpolation

17. Caroline Mendonça Costa ParMETIS and Hypre Page 17 Large Linear Systems PCG PETSc - Portable, Extensible Toolkit for Scientific Computation Freely distributed at http://www.mcs.anl.gov/petsc/index.html

18. Caroline Mendonça Costa ParMETIS and Hypre Page 18 Preconditioning Hypre – High Performance Preconditioners BoomerAMG Offered together with PETSc

19. Caroline Mendonça Costa ParMETIS and Hypre Page 19 Preconditioning BoomerAMG: Parallel Algebraic Multigrid ● General parallelization can be extended from GMG ● Except for coarse-grid selection ● Naturally sequential ● Parallelization of the coarse-grid selection ● Parallel CLJP algorithm ● Parallel Ruge-Stüben coarsening ● Parallel Falgout coarsening

20. Caroline Mendonça Costa ParMETIS and Hypre Page 20 Preconditioning BoomerAMG: Parallel Algebraic Multigrid ● Parallel CLJP algorithm ● Based on Luby's graph coloring algorithm ● Coarsening is computed for each color separated ● Poor coarsening on the interior of processors domains ● Parallel Ruge-Stüben coarsening ● Nodes with maximal weight are included in the coarse grid ● Each processor applies RS to its own nodes subset ● Difficulties on the boundaries of processors domains ● Parallel Falgout coarsening ● Uses CLJP near boundary nodes of each processor ● Uses RS on the internal nodes of each processor

21. Caroline Mendonça Costa ParMETIS and Hypre Page 21 A Few Results... Table 1. Solution time Unable to isolate the setup time of the AMG RS tend to behave as Falgout as the number of processor increase? Setup time is included in the final solution time Falgout leads to a better conditioned system than RS and CLJP or it takes less time to perform coarsening than RS and CLJP?

22. Caroline Mendonça Costa ParMETIS and Hypre Page 22 More to come... ● Isolate setup and solution time of the AMG ● Show more results ● Using different unstructured and larger meshes ● Using more processors ● Using different interpolation types ● Using different smoothers

23. Caroline Mendonça Costa ParMETIS and Hypre Page 23 Some bibliography ● ParMetis ● Manual: http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download ● Parallel Multilevel k-way Partitioning Scheme for Irregular Graphs: http://glaros.dtc.umn.edu/gkhome/node/109 http://glaros.dtc.umn.edu/gkhome/node/109 ● Hypre ● Manual: https://computation.llnl.gov/casc/hypre/software.html https://computation.llnl.gov/casc/hypre/software.html ● BoomerAMG: https://computation.llnl.gov/casc/linear_solvers/pubs/boom.pdf https://computation.llnl.gov/casc/linear_solvers/pubs/boom.pdf

24. Caroline Mendonça Costa ParMETIS and Hypre Page 24 Thank you for your attention!