1. Avoiding Communication in Sparse Iterative Solvers Erin Carson Nick Knight CS294, Fall 2011

2. Motivation: Sparse Matrices Many algorithms for scientific applications require solving linear systems of equations: Ax = b 2 Figure: Simulating Pressure over Airfoil. Source: http://www.nada.kth.se In many cases, the matrix A is sparse – Sparse matrix: a matrix with enough zero entries to be worth taking advantage of This means that information is “local” instead of “global”. A given variable only depends on some of the other variables. – Example: Simulating Pressure around Airfoil

3. Sparse vs. Dense Linear Algebra Different performance considerations for sparse matrices – Low computational intensity (small # non-zeros per row) – sparse matrix operations communication bound – Sparse data structures (CSR, diagonal, etc.) – Matrix structure encodes dependencies: sparsity pattern gives additional opportunities for avoiding communication Changes the way we must think about partitioning – Finding an optimal partition is an NP-complete problem, heuristics must be used Order of execution matters: must consider fill-in Run-time auto-tuning important – pOSKI (Parallel Optimized Sparse Kernel Interface)

4. Sparse Solvers in Applications 4 Figure: Contour Detection [CSNYMK10] Image Processing Applications – Ex: Image segmentation, Contour detection Physical simulations – Solving PDEs Often used in combination with Multigrid as bottom-solve Ex: Simulating blood flow (Parlab’s Health App) Mobile/Cloud applications – Communication more important where bandwidth is very limited, latency is long (or if this parameters are variable between machines!) Auto-tuning becomes more important if we don’t know our hardware Figure: ParLab Health App: Modeling Blood Blow in the Brain

5. Solving a Sparse Linear System Iterative methods iteratively refine an approximate solution to the system – Used when System is large and sparse – direct method too expensive We only need an approximation – don’t need to solve exactly, so less operations needed A is not explicitly stored – Ex: Krylov Subspace Methods (KSMs) 5 = x A L U Direct methods solve a linear system in a finite sequence of operations – Often used to solve dense problems – Ex: Gaussian Elimination Direct Method for Solving Ax = b Initial guess Convergence? Return solution Yes No Refine Solution Iterative Method for Solving Ax = b

6. 0 r K How do Krylov Subspace Methods Work? 6 In each iteration, – Sparse matrix-vector multiplication (SpMV) with A to create new basis vector Adds a dimension to the Krylov Subspace – Use vector operations to choose the “best” approximation of the solution in the expanding Krylov Subspace (projection of a vector onto a subspace) How “best” is defined distinguishes different methods Examples: Conjugate Gradient (CG), Generalized Minimum Residual Methods (GMRES), Biconjugate Gradient (BiCG) proj K (r)

7. Communication in SpMV v1v2 v3v4

8. Hypergraph Model for Communication in SpMV v1v2 v3v4 v1v2 v3v4 Pattern Matrix Graph Representation Hypergraph Representation (Column-Net Model)

9. Graph vs. Hypergraph Partitioning Consider a 2-way partition of a 2D mesh: The cost of communicating vertex A is 1 – we can send the value in one message to the other processor According to the graph model, however the vertex A contributes 2 to the total communication volume, since 2 edges are cut. The hypergraph model accurately represents the cost of communicating A (one hyperedge cut, so communication volume of 1. Unlike graph partitioning model, the hypergraph partitioning model gives exact communication volume (minimizing cut = minimizing communication) Edge cut = 10 Hyperedge cut = 7

10. Example: Hypergraph vs. Graph Partitioning 2D Mesh, 5-pt stencil, n = 64, p = 16 Hypergraph Partitioning (PaToH) Total Comm. Vol = 719 Max Vol per Proc = 59 Graph Partitioning (Metis) Total Comm. Vol = 777 Max Vol per Proc = 69

11. Summary