1. ParFUM Parallel Mesh Adaptivity Nilesh Choudhury, Terry Wilmarth Parallel Programming Lab Computer Science Department University of Illinois, Urbana Champaign

2. Parallel Framework for Unstructured Meshes (ParFUM) Parallel scalable framework for Unstructured Mesh computations. Handles the parallel geometry details Leaves physics and numerics to the user Provides several advanced features: Parallel Mesh refinement and coarsening Dynamic automatic load balancing Collision detection, visualization, etc Supports ghost layers (based on nodes or elements) Presents a clean and “almost serial” interface Clean language bindings for F90 and C

3. Features Originally designed for standard structural dynamics codes Extended to support Fluid Dynamic codes(Finite Volume) Local element/node ID numbering Arbitrary element types, and arbitrary user data associated to them Boundary condition representation for faces, edges and nodes Efficient ID translation for communication Partitioning (Metis and ParMetis based) Field registration and updating for shared nodes, ghosts 3-D Fractography in FEM Rocket Burn Simulation, CSAR

4. Features (contd.) General purpose migration and load balancing Fully migratable threads implemented Minor changes to make code migratable Framework handles ghost layer generation along processor boundaries Could be based on problem symmetries Incremental update of ghosts Mesh adaptivity operations: Refine(mesh, region_to_refine, target_sizes) Coarsen(mesh, region_to_refine, target_sizes) A mesh with two chunks Edge based 1-based ghosts Node based 1-layer ghosts

5. Parallel Mesh Adaptivity Support Independent mesh adaptivity operations in parallel Locking is node based Chunk based locking available too No global synchronization Adjacency data structures: Node-to-node Element-to-element Node-to-element Element-to-node Mesh Adjacency e2e,e2n,n2e,n2n Generate, Modify … Mesh Modification Lock(),Unlock() Add/Remove Node() Add/Remove Element() Nodes: Local, Shared, Ghost Elements: Local,Ghost Application (serial or parallel) Mesh Adaptivity Edge Flip, Edge Bisect, Edge Contract, Longest edge bisect, … ParF UM

6. Primitive Adaptivity Operations Mesh Modification operations are implemented in terms of primitive operations Add_node Remove_node Add_element Remove_element Lock_Node UnLock_Node Primitive operations must: Perform the operation only on the local and all applicable remote processors Do all node and element number translations through IDXL lists Update processor boundaries and nodes/elements on all applicable processors

7. Atomic Adaptivity Operations To the higher level adaptive algorithms, we make available the following atomic operations: Edge flip Edge bisect Edge contract Longest edge bisect These operations are implemented in terms of the primitive mesh adaptivity operations, handle locking and ensure no race conditions Atomic – any virtual chunk can call these operations on any set of nodes in any order

8. Node_Lock/UnLock Node_Lock(mesh, nodeId, isReadLock) Node_UnLock(mesh, nodeId, isReadLock) For a local node lock that node on the requesting chunk For a shared node Find all the chunks that share it Find the chunk with the lowest chunkId Lock the node on that chunk (that is the unique lock) Minimizes communication overhead For a ghost node Decide which remote chunk it belongs to If belongs to a set a remote chunks, decide on the smallest remote chunk on them and lock it there

9. Add_Node/Remove_Node Add_Node(mesh, *adjacentNodes, numAdjacentNodes, chunkNo): Find an unused node or create a new one Update all adjacencies for that node Update IDXL list entries Might be a local, shared or ghost node Interpolate the attributes on the new node Remove_Node(mesh, nodeId): Decide which chunk it belongs to Reset adjacencies Update all entries in the IDXL lists Remove the node

10. Add_Element Add_Element(mesh, *conn, connSize, elemType, chunkNo) These operations are designed to work in any number of dimensions Can force it to add element to 'chunkNo'; -1 otherwise Based on 'conn' decide which chunk it should belong to! An element could be a ghost: if any node is a ghost all nodes are shared Add element as a local elem on only one chunk Ghosts maintained are 1-away node ghosts Add node ghosts and element ghosts on a number of chunks because of this element addition A ghost node might need to be converted to a shared node Update adjacencies on all neighboring nodes/elements on this and neighboring chunks

11. Remove_Element Remove_Element(mesh, elemId, elemType, permanent) An element can belong to only one chunk Easy to decide where it belongs to from elemId and IDXL tables 'permanent' implies to lose ghosts (nodes/elements) Some operations are guaranteed to not change chunk boundaries To optimize them – skip ghost modifications by setting this flag Might have to convert a shared node to a ghost node Delete ghost nodes/elements on remote chunk Sender of a ghost always decides if a ghost needs to be removed on a chunk Only if a node is being removed, shared/ghost, it will lose ghost elements Update adjacencies on all neighboring nodes/elements on this and neighboring chunks

12. Mesh Modification: Edge_flip Edge Flip: Lock required nodes Test feasibility of flip Remove elements e1 Remove element e2 Add element (n1,n2,n4) Add element (n2,n3,n4) Unlock nodes

13. Mesh Modification: Edge_bisect Edge Bisect: Lock required nodes Test feasibility of bisect Remove elements e1 Remove element e2 Add node n5 & interpolate Add element (n1,n2,n5) Add element (n3,n5,n2) Add element (n4,n5,n3) Add element (n4,n1,n5) Unlock nodes

14. Mesh Modification: Edge_contract Edge Contract: Lock required nodes Test feasibility of contract Remove elements e1 Remove element e2 Update coordinates of n1 for i=6 to 9 Remove ei Add ei' (conn. n1 instead of n2) end Remove n2 Unlock nodes

15. Mesh Modification in Parallel Mesh on Processor 1 before edge flip Mesh on Processor 2 before edge flip Mesh on Processor 1 after edge flip Mesh on Processor 2 after edge flip

16. Edge_flip ● These are the next layer of operations, which are dimension specific (2D operations) ● All these operations are on two nodes, which represent an edge ● Edge_flip(int node1, int node2) ● Find the two elements, e1(n1,n2,n3) / e2(n1,2,n4) adjacent to this edge ● Lock nodes n1,n2,n3,n4 ● If they are not on the same chunk ● Delete the element on the remote chunk ● Force add it to the local chunk ● Delete e1(n1,n2,n3) and e2(n1,n2,n4) ● Add e1'(n1,n3,n4) and e2'(n2,n3,n4) ● Unlock nodes n1,n2,n3,n4

17. Edge_bisect ● Edge_bisect(int node1, int node2) ● Find the two elements, e1(n1,n2,n3) / e2(n1,2,n4) adjacent to this edge ● Lock nodes n1,n2,n3,n4 ● If while trying to lock, the edge no longer exists, bail out! ● Another operation modified the edge between the time the operation was decided till it was about to be done ● Doesn't matter if e1/e2 are not on same chunk ● Delete e1(n1,n2,n3) and e2(n1,n2,n4) ● Add node n5 (“between” n1 and n2) ● Lock n5 ● Add e1'(n1,n3,n5) ● Add e2'(n2,n3,n5) ● Add e3'(n1,n4,n5) ● Add e4'(n2,n4,n5) ● Unlock nodes n1,n2,n3,n4,n5 ● Return the new node

18. Longest_Edge_Bisect ● Longest_Edge_Bisect(int e1) ● Find longest edge of e1 – say n1,n2 ● Find the element, e2(n1,n2,n4) adjacent to this edge ● If n1,n2 is a mesh boundary, return edge_bisect(n1,n2) ● Find e2's longest edge ● If it is (n1,n2), then return edge_bisect(n1,n2) ● Else newnode=edge_bisect(n1,n2) ● Recursively find the neighbor on the longest edge of e2 ● Propagate the refinement on it ● edge_flip(newnode,n4) ● Return the new node

19. Edge_contract ● Edge_contract(int node1, int node2) ● Find the two elements, e1(n1,n2,n3) / e2(n1,2,n4) adjacent to this edge ● Lock nodes n1,n2,n3,n4 ● If while trying to lock, the edge no longer exists, bail out! ● Lock nodes of all elements connected to n1 and n2 ● Doesn't matter if e1/e2 or any of the other elements are ondifferent chunks ● Decide which of n1 or n2 needs to be deleted ● Compute the new attributes of “keepnode” ● Check if any of the new elements created would “flip”. If yes, bail out! ● Delete e1(n1,n2,n3) and e2(n1,n2,n4) and all elements connected to “deletenode” ● Update the connectivity of all deleted elements except e1/e2 to use “keepnode” instead of “deletenode” and add those elements ● Delete “deletenode” ● Unlock all nodes which were locked for this operation ● Return “keepnode”