1. Parallelizing Gauss-Seidel Solver/Pre-conditioner Aim: To parallelize a Gauss-Seidel Solver, which can be used as a pre-conditioner for the finite element code poicyc.cpp Parallelizing a GS solver involves two steps: 1.Re-arrange the stiffness matrix to permit a parallel solution 2.Solve the re-arranged stiffness matrix on multiple CPUs After parallelizing: A. Interface the above with poicyc.cpp to solve large finite element problems on multiple CPUs B. The Gauss-Seidel solver can be extended to SOR solver with the inclusion of factor omega and interpolating with previous vector

2. Part 1: Re-ordering the Stiffness Matrix Why: The stiffness matrix needs to be re-ordered such that its main diagonal comprises of diagonal sub-matrices. This is achieved by re-numbering the nodes using a ‘coloring scheme’. The new re- numbered mesh will have the structure on the right. Stiffness Matrix Before Re-ordering Stiffness Matrix After Two-color Re-ordering

3. Node Coloring and Re-ordering General Procedure: For each node in the mesh connectivity, search for all its neighbours. This includes all nodes in all the elements that this node is connected to. Assign a color (i.e. a numerical label) to each node one by one, with the condition that a node cannot have the same color as any of its neighbouring nodes. Structure of the Code: For each node, check neighbours, and store all those nodes For each node, assign color ‘0’. Then assign the smallest numerical label that its neighbours does not have. Collect all nodes of a color, and number them in numerically increasing order. Do the same for all nodes of all colors. Re-order the element connectivity based on the new numbering scheme. Create new vectors for ‘p’ (coordinate data) and ‘t’ (connectivity).

4. Images of the Mesh and Stiffness Matrices after Re-ordering FE mesh of the cube (uses 10-node tetrahedrons) Nodal coloring of a 2-D 6-node Triangular mesh

5. Stiffness Matrix After Re-ordering (for the cubic mesh in previous slide) FE mesh of the cube (uses 10-node tetrahedrons) Note multiple Diagonal ‘sub-matrices’ along main Diagonal-one for each color 13 Colors required above

6. Stiffness Matrix After Re-ordering for a 25,000/Half-million node mesh

7. Solving the New Stiffness Matrix by Gauss-Seidel Gauss-Seidel Solution for: Ax=B 1.Start with a trial vector x 1 (=B) Compute: B_new = B - U x 1 2. Solve: [D + L] = B_new 3. Update x & repeat Steps 1 through 3 With re-ordered matrix, we have a parallel solution: 1.All rows in step 1 can be solve simultaneously (parallel) 2.Forward substitution can be parallelized thus: 1.Solve D_1 = x _UPPER HALF (simple division: x(i)= b(i)/a(i) ) 2.Compute y = B_new –L x _UPPER HALF 3.Solve x_2 = D_2 /y 3.For multiple colors, do step 1, then repeat steps 2 and 3 for each color U L D_1 D_2

8. Parallelizing the GS code (on two processes) Step 1 Cyclically, submit each row to a process to Compute RHS – U*x Gather all rows, assemble B-new and broadcast to All procs. About half the rows are Computed by process 1 Send Computed Rows to Proc. 0 Step 2: Compute solution To diagonal elements of Each sub-matrix (on proc. 0) Step 3: Cyclically submit Each row to a process and Compute: y = B-Lx x= y/d Gather all rows, assemble B-new and broadcast to All procs. About half the rows are Computed by process 1 Send Computed x to Proc. 0 Iterate among These steps B/Cast B-new For all colors B/Cast x

9. Run-Times on various processors 25,000 node model 500,000 node model For 300 iterations 1 p2 p4 p 27.2 s50.7 s 9.2 s 9m 48 s19m 33s Node coloring: 2-3 s 17 s