1. The Charm++ ParFUM Framework PARallel Framework for Unstructured Meshing presented by: Isaac Dooley Parallel Programming Lab University Illinois Urbana-Champaign http://charm.cs.uiuc.edu

2. ParFUM Overview

3. Charm Workshop 20053 Why use the ParFUM Framework? Publicly Available (included with Charm++) Makes parallelizing a serial code faster and easier Handles mesh partitioning Handles communication Handles load balancing (via Charm++) Bindings for Fortran, C++, C Provides Advanced features Parallel Adaptivity and Incremental Mesh Modification Visualization Support with ParaView or NetFEM Collision Detection Library Iterative Matrix Solver Library (IFEM)

4. Charm Workshop 20054 Features Overview Originally Designed for Finite Element/Volume Problems Parallel Partitioning Field registration and updating for shared nodes, ghosts -- “User can own Data” Arbitrary element types, and arbitrary user data associated to elements or nodes Boundary condition representation for faces, edges and nodes Efficient ID translation mechanism for communication Mesh Modification Support 3-D Fractography in ParFUM Rocket Burn Simulation, CSAR

5. Charm Workshop 20055 ParFUM Framework Users CSAR Rocflu: Fluids solver RocRem: Remeshing / Parallel Solution Transfer CPSD SpaceTime meshing Frac3D Fracture Mechanics Dendritic Growth Metal Solidification process

6. Charm Workshop 20056 Outline for the rest of this talk ParFUM concepts & philosophy Program Structure Synchronization of Ghost & Shared entities Topological Relationships (Adjacencies) Parallel Mesh Adaptivity Obtaining & Compiling ParFUM Libraries within ParFUM Collision Detection Visualization Tools One Sample Program

7. ParFUM Concepts

8. Charm Workshop 20058 ParFUM Basics Nodes Points in the problem domain’s space. Elements Discretized areas or volumes representing portions of a problem domain Arbitrary Dimension and Type Each element is defined by a set of nodes Data Attributes, can be associated with nodes or Elements Data can be owned by either Framework or Application

9. Charm Workshop 20059 Serial Mesh N5N4N2E3 N4N2N1E2 N4N3N1E1 Surrounding Nodes Element

10. Charm Workshop 200510 Partitioning Partition the ParFUM Mesh into multiple chunks Distribute elements, replicate shared nodes and/or add ghosts Keep track of communication Partition so that communication is minimized

11. Charm Workshop 200511 Partitioned Mesh N3N2N1E2 N4N3N1E1 Surrounding Nodes Element N3N2N1E1 Surrounding Nodes Element Shared Nodes N3N4 N1N2 BA

12. Charm Workshop 200512 ParFUM Parallel Model: Shared Nodes “Shared Node” model Element computations based on values of surrounding nodes Node values are sum of surrounding elements Example: Mechanical Simulation Element stresses are computed from element’s nodal displacements

13. Charm Workshop 200513 ParFUM Parallel Model: Ghosts “Ghost” model Element computations based only on values of surrounding nodes and elements Example: Fluid Dynamics Element pressures and velocities come from neighboring element pressures and velocities, and node locations

14. Charm Workshop 200514 Virtualization Charm++ Runtime System Applications built using migratable objects Each ParFUM mesh chunk mapped to a migratable object. Virtualization = multiple migratable objects per processor Load Balancing High(90-100%) Processor Utilization and Scaling. System Implementation User View

15. Charm Workshop 200515 ParFUM Application On Eight Physical Processors Benefit of Virtualization to Structural Dynamics Applications

16. ParFUM Program Structure

17. Charm Workshop 200517 Consists of at least two user-written subroutines init driver init is called on chunk 0 driver is called on every chunk Structure of an ParFUM Application

18. Charm Workshop 200518 init() subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine

19. Charm Workshop 200519 driver() subroutine driver get local mesh chunk time loop ParFUM computations communication more ParFUM computations end time loop end subroutine

20. Charm Workshop 200520 Structure of an ParFUM Application init() Update driver

21. Charm Workshop 200521 Framework Owns Data User gives framework mesh in init using FEM_Mesh_Data() User Accesses mesh by calling FEM_Mesh_Data() in driver to copy data into the user’s arrays User Owns Data User registers mesh with framework in init using FEM_Register_entity() and FEM_Register_array() User uses their own data arrays in driver User must also supply a resize function, to be called when framework partitions mesh and resizes mesh in adaptive refinement calls. Data Management In ParFUM

22. Charm Workshop 200522 void Mesh_data( int mesh, int entity, int attr, void *data, int first, int length, int datatype, int width ); Get/Set, and multi-mesh support NODE, ELEM, SPARSE (+GHOST) DATA, CONN, SYM,GLOBALNO,… User data: width x length array Apply to fir st…first+length-1 User data formatting Setting/Getting Mesh Data

23. Charm Workshop 200523 Mesh_data( FEM_Mesh_default_read() FEM_Mesh_default_read(), FEM_NODE, FEM_DATA+23, coord, 0,nNodes, FEM_DOUBLE,3 ); Mesh Access: Example 1 Read from default mesh Access nodal data Some user defined attribute #23 User data: 3 x nNodes array Get data for nodes 0 to nNodes 3 doubles for each node

24. Charm Workshop 200524 Mesh_data( FEM_Mesh_default_write() FEM_Mesh_default_write(), FEM_ELEM, FEM_CONN, conn, 0,nElem, FEM_INDEX_0, 3 ); Mesh Access: Example 2 Write to default mesh Modify element data Element Connectivity Connectivity Array: 3 x nElem For elements 0 to nElem 3 integers for each element (0-based C-style indexing)

25. Charm Workshop 200525 Mesh_data( FEM_Mesh_default_read() FEM_Mesh_default_read(), FEM_ELEM+FEM_GHOST, FEM_CONN, conn, 0,nElem, FEM_INDEX_0,3 ); Mesh Access: Example 3 Read from default mesh Access ghost element data

26. ParFUM Ghost Elements

27. Charm Workshop 200527 Ghost Elements: Overview Most ParFUM programs communicate via shared nodes Some computations require read-only copies of remote elements—“ghosts” Stencil-type finite volume computation Many kinds of mesh modification

28. Charm Workshop 200528 Ghosts: 2D Example 123412 Ghost of 3 Ghost of 2 34 Serial Mesh Left Chunk Right Chunk

29. Charm Workshop 200529 Defining Ghost Layers: Add ghost elements layer-by-layer from init A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes For 2D, a tuple might be a 2-node edge For 3D, a tuple might be a 4-node face You specify a ghost layer with Add_ghost_layer(tupleSize,ghostNodes) ghostNodes indicates whether to add ghost nodes as well as ghost elements.

30. Charm Workshop 200530 Ghosts: Node adjacency 0 1 2 /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1,0); /* 1 node per tuple */ const static int tri2node[]={0,1,2}; FEM_Add_ghost_elem(0,3,tri2node);

31. Charm Workshop 200531 Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */ const static int tri2edge[]={0,1, 1,2, 2,0}; FEM_Add_ghost_elem(0,3,tri2edge); 0 1 2

32. ParFUM Ghosts & Shared Node Synchronization

33. Charm Workshop 200533 Node Fields Framework handles combining data for shared nodes and keeps them in sync Framework does not understand meaning of node fields, only their location and types Framework needs to be informed of locations and types of fields Create_field once, Update_field every timestep

34. Charm Workshop 200534 Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) :: datatype, len

35. Charm Workshop 200535 Update Field: Shared Nodes subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes Can be used to sum forces contributed to a node in FE Method

36. Charm Workshop 200536 Update Field: Ghosts subroutine Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts Can be used to update ghost copies of local nodes(with current displacements) in FE Method

37. Topological Relationships: Adjacencies

38. Charm Workshop 200538 Adjacencies User registers element to node connectivity Framework can derive other relationships: Element to Element Node to Node Node to Element User specifies a “tuple” to specify when two arbitrary elements are adjacent. The framework uses this to construct the other adjacencies if needed.

39. Charm Workshop 200539 Adjacencies const int triangleFaces[6] = {0,1,1,2,2,0}; FEM_Add_elem2face_tuples(mesh,0,2,3,triangleFaces); FEM_Mesh_create_elem_elem_adjacency(mesh); FEM_Mesh_create_node_elem_adjacency(mesh); FEM_Mesh_create_node_node_adjacency(mesh); User must explicitly have framework generate the adjacency tables

40. Charm Workshop 200540 Adjacencies Get a list of elements adjacent to element e: e2e_getAll(int e, int *neighbors); Get a list of elements adjacent to node n: n2e_getAll(int n, int **adjelements, int *sz); The adjacency tables are created as attributes which can be accessed via Mesh_Data() or by simple accessor functions.

41. Charm Workshop 200541 Adjacencies Caveats or Limitations: Currently some of the adjacency functions only work correctly for meshes containing one type of element. Two nodes are adjacent if and only if the two nodes are listed in the connectivity for some element. Thus in a rectangular grid, two nodes on opposite corners of a rectangle will be listed as adjacent nodes.

42. How Do I get a copy of ParFUM and Run a ParFUM Application?

43. Charm Workshop 200543 ParFUM or FEM? FEM is the old name for ParFUM FEM and ParFUM are currently the same library in Charm++ The FEM framework as described in the manual does not contain some new features: Adjacency Data Structures, Adaptivity, Mesh Modification The codebase currently uses the old naming conventions of FEM, functions start with “FEM_” The current version of FEM/ParFUM can no longer run apart from Charm++

44. Charm Workshop 200544 Where to Get It ? ParFUM is included in Charm++ CVS distribution CSH: setenv CVSROOT ":pserver:checkout@charm.cs.uiuc.edu:/cvsroot" Or BASH: export CVSROOT=":pserver:checkout@charm.cs.uiuc.edu:/cvsroot" You should now be able to do a > cvs login (no password needed, just type [Enter] at prompt) and then > cvs co -P charm ParFUM-FEM is in charm/src/libs/ck-libs/fem

45. Charm Workshop 200545 How to Build It ? > cd charm and do >./build LIBS net-linux -O This will make a net-linux directory, with bin, include, lib etc subdirectories.

46. Charm Workshop 200546 How to Compile & Link ? Use “charmc”: available under bin a multi-lingual compiler driver, understands f90 Knows where modules and libraries are Portable across machines and compilers Linking use “-language femf” : for F90 Use “–language fem” : for C/C++ See example Makefiles charm/examples/fem/…

47. Charm Workshop 200547 How to Run ? Charmrun A portable parallel job execution script Specify number of processors: +pN Specify number of chunks: +vpN Special “nodelist” file for net-* versions./charmrun./pgm +vp100 +p16

48. Charm Workshop 200548 Example./charmrun pgm +p4 +vp70 Nodelist File: $(HOME)/.nodelist group main ++shell ssh host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.edu host tur0004.cs.uiuc.edu

49. ParFUM Parallel Mesh Adaptivity

50. Charm Workshop 200550 Mesh Adaptivity in ParFUM 2D refinement 2D coarsening 3D refinement 3D coarsening -- coming soon Support for user defined mesh operations.

51. Charm Workshop 200551 Mesh Modification Primitives ParFUM provides incremental asynchronous parallel mesh modificiation primitives: Add_node() Remove_node() Add_element() Remove_element() Higher Level Operations are built using these primitives edge_bisect() edge_flip() edge_collapse() …

52. Mesh Adjacency e2e,e2n,n2e,n2n Generate, Modify … Mesh Modification Lock(),Unlock() Add/Remove Node() Add/Remove Element() Nodes: Local, Shared, Ghost Elements: Local,Ghost SDG Application API (serial or parallel) ParFUM Structure Mesh Adaptivity Edge Flip, Edge Bisect, Edge Contract, … ParFUM

53. Charm Workshop 200553 Mesh Modification Examples Edge Flip: Remove elements e1 Remove element e2 Add element (n1,n2,n4) Add element (n2,n3,n4)

54. Charm Workshop 200554 Mesh Modification Examples Edge Bisect: Remove elements e1 Remove element e2 Add node Add element (n1,n2,n5) Add element (n3,n5,n2) Add element (n4,n5,n3) Add element (n4,n1,n5)

55. Charm Workshop 200555 Mesh Modification in Parallel Mesh on Processor 1 before edge flip Mesh on Processor 2 before edge flip Mesh on Processor 2 after edge flip

56. Charm Workshop 200556 Mesh Modification in ParFUM Primitive Operations must do the following: Perform the operation on local and all applicable remote processors Convert local nodes to shared nodes when they become part of the new boundary Update ghost layers(nodes and elements) for all applicable processors. The ghost layers can grow or shrink

57. Charm Workshop 200557 Parallel Mesh Refinement/Coarsening Parallel Refinement and Coarsening algorithms in action: A wave propagates through a block across 8 processors

58. Charm Workshop 200558 Parallel 3D Refinement Built from same primitives Longest edge based refinement Longest face based refinement Currently implementing 3D coarsening

59. Charm Workshop 200559 Refinement and Coarsening in a 2D ParFUM Application Shock propagation and reflection down the length of the bar Adaptive mesh modification to capture the shock propagation

60. Charm Workshop 200560 Solution Comparison Initial Mesh Adaptive Mesh Fine Mesh

61. ParFUM Collision Detection

62. Charm Workshop 200562 Charm++ Collision Detection Detect collisions/intersections/contacts between objects scattered across processors Built on Charm++ Arrays Overlay regular 3D sparse grid of voxels (boxes) Send objects to all voxels they touch Collect collisions from each voxel Collision response is left to caller

63. ParFUM Mesh Visualization

64. Charm Workshop 200564 Mesh Visualization in ParFUM Currently we support: NetFEM Online Visualization NetFEM Offline Visualization ParaView Offline Visualization

65. Charm Workshop 200565 NetFEM Client: pretty pictures Wave dispersion off a crack (simplified frac3d)

66. Charm Workshop 200566 ParFUM Unified Visualization Interface n=NetFEM_Begin(…); NetFEM_Nodes(…); NetFEM_Vector(…); NetFEM_Elements(…); NetFEM_Scalar(…); NetFEM_End(n); Start output for one timestep, specifying Output method(online/offline) Specify number of nodes and coord. Give an array of nodal vector data Give a second array of nodal data Specify number of elements and conn. Give an array of scalar element data Give a second array of element data Give a third array of element data NetFEM_End(n);

67. Charm Workshop 200567 NetFEM n=NetFEM_Begin(2,t,NetFEM_POINTAT); NetFEM_Nodes(n,nnodes,(double *)g.coord,"Position (m)"); NetFEM_Vector(n,(double *)g.d,"Displacement (m)"); NetFEM_Vector(n,(double *)g.v,"Velocity (m/s)"); NetFEM_Elements(n,nelems,3,(int *)g.conn,"Triangles"); NetFEM_Scalar(n,g.S11,1,"X Stress (pure)"); NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)"); NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)"); NetFEM_End(n); ParFUM Unified Visualization Interface

68. Charm Workshop 200568 NetFEM Online Visualization To allow the NetFEM client to connect, you add NetFEM registration calls to your server Register nodes and element types Register data items: scalars or spatial vectors associated with each node or element You provide the display name and units for each data item Link your program with “-module netfem” Run with “++server”, and connect!

69. Charm Workshop 200569 NetFEM: Setup n=NetFEM_Begin(FEM_My_partition(),timestep, dim,NetFEM_POINTAT) Call this each time through your timeloop; or skip timestep identifies this data update dim is the spatial dimension—must be 2 or 3 Returns a NetFEM handle n used by everything else NetFEM_POINTAT for online visualization NetFEM_End(n) Finishes update n

70. Charm Workshop 200570 NetFEM: Nodes NetFEM_Nodes(n,nnodes,coord,”Position (m)”) Registers node locations with NetFEM—future vectors and scalars will be associated with nodes n is the handle returned by NetFEM_Begin nnodes is the number of nodes coord is a dim by nnodes array of doubles The string describes the coordinate system and meaning of nodes Currently, there can only be one call to nodes

71. Charm Workshop 200571 NetFEM: Node Displacement

72. Charm Workshop 200572 NetFEM: Elements NetFEM_Elements(n,nelem,nodeper, conn,”Triangles”) Registers elements with NetFEM—future vectors and scalars will be associated with these elements n is the handle returned by NetFEM_Begin nelem is the number of elements nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices The string describes the kind of element Repeat to register several kinds of element Perhaps: Triangles, squares, pentagons, …

73. Charm Workshop 200573 NetFEM: Element Stress

74. Charm Workshop 200574 NetFEM: Vectors NetFEM_Vector(n,val,”Displacement (m)”) Registers a spatial vector with each node or element Whichever kind was registered last n is the handle returned by NetFEM_Begin val is a dim by nitems array of doubles There’s also a more general NetFEM_Vector_field in the manual The string describes the meaning and units of the vectors Repeat to register multiple sets of vectors Perhaps: Displacement, velocity, acceleration, rotation, …

75. Charm Workshop 200575 NetFEM: Element Stress

76. Charm Workshop 200576 NetFEM: Element Velocity

77. Charm Workshop 200577 NetFEM: Zoom in

78. Charm Workshop 200578 NetFEM: Outline Elements

79. Charm Workshop 200579 Offline Visualization with NetFEM Call NetFEM_Begin() with NetFEM_WRITE, ParFUM application will then write out binary output dump files to a new directory called “NetFEM” containing directories for each timestep Visualize with command netfem NetFEM/10 In offline mode, the ``update'' button fetches the next extant timestep directory.

80. Charm Workshop 200580 Offline Visualization with ParaView The directory NetFEM can be converted to a ParaView/VTK compatible XML format. Converter can be built by calling “make” in charm/tmp/libs/ck-libs/netfem/ParaviewConverter/ Run the resulting program NetFEM_To_Paraview to convert the NetFEM directory to a new directory called ParaViewData. ParaView can open the files inside this directory either as individual chunks, or as the entire mesh. Scan through the timesteps using the standard timestep slider.

81. Charm Workshop 200581 Offline Visualization with ParaView Limitations: Multiple element types with varying numbers of data attributes may not work Current converter produces ASCII XML files, not binary, so they use excessive disk space. Benefits: Advanced visualization functionality More stable than offline NetFEM which crashes if you try to access non-existent timesteps, or huge meshes

82. Migration and Load Balancing

83. Charm Workshop 200583 Advanced API: Migration Chunks may not be computationally equal: Results in load imbalance Multiple chunks per processor Chunks cannot have writable global data Automatic load balancing Migrate chunks to balance load How to migrate allocated data for chunks ? Embed it in a user-defined type

84. Charm Workshop 200584 Chunk Data Example MODULE my_block_mod TYPE my_block INTEGER :: n1,n2x,n2y REAL*8, POINTER, DIMENSION(:,:) :: arr END TYPE END MODULE

85. Charm Workshop 200585 Pack/Unpack (PUP) Routine SUBROUTINE pup_my_block(p,m) USE my_block_mod USE pupmod INTEGER :: p TYPE(my_block) :: m call fpup_int(p,m%n1) call fpup_int(p,m%n2x) call fpup_int(p,m%n2y) IF (fpup_isUnpacking(p)) THEN ALLOCATE(m%arr(m%n2x,m%n2y)) END IF call fpup_doubles(p,m%arr,m%n2x*m%n2y) IF (fpup_isDeleting(p)) THEN DEALLOCATE(m%arr) END IF END SUBROUTINE

86. Charm Workshop 200586 Registering Chunk Data !- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p,m) use my_block_mod INTEGER :: p TYPE(my_block) :: m end subroutine end interface TYPE(my_block) :: m CALL FEM_Register(m,pup_my_block)

87. Charm Workshop 200587 Migration Every chunk driver calls FEM_Migrate() Framework calls PUP for getting the size of packed data For packing data Chunk migrates to new processor Framework calls PUP for unpacking Driver returns from FEM_Migrate()

88. A sample ParFUM Program

89. Charm Workshop 200589 Simple2D Can be found in charm/examples/fem/simple2D Generates simple but interesting visualizations A constrained triangle Finite Element Code Here we will show the init and driver routines, almost exactly copied from pgm.C, omitting some comments and trivial code fragments.

90. Charm Workshop 200590 extern "C" void init(void) { CkPrintf("init started\n"); double startTime=CmiWallTimer(); int nPts=0; //Number of nodes vector2d *pts=0; //Node coordinates int nEle=0; connRec *ele=NULL; // Omitted for readability: // Read in mesh from users files to ele and pts // Tell framework we will be writing to the mesh int fem_mesh=FEM_Mesh_default_write();

91. Charm Workshop 200591 FEM_Mesh_data( fem_mesh, // Add nodes to the current mesh FEM_NODE, // We are registering nodes FEM_DATA+0, // Register the point locations which (double *)pts, // The array of point locations 0, // First node is 0 nPts, // The number of points FEM_DOUBLE, // Coordinates are doubles 2); // Points have dimension 2 (x,y) FEM_Mesh_data( fem_mesh, // Add elements to the current mesh FEM_ELEM+0, // Element type 0 FEM_CONN, // Register the connectivity table for this (int *)ele,// The array of point locations 0, // 0 based indexing nEle, // The number of elements FEM_INDEX_0,// We use zero based node numbering 3); // Elements have three nodes

92. Charm Workshop 200592 delete[] ele; delete[] pts; CkPrintf("Finished with init\n"); }

93. Charm Workshop 200593 extern "C" void driver(void){ int nnodes,nelems,ignored; int i, myId=FEM_My_partition(); myGlobals g; FEM_Register(&g,(FEM_PupFn)pup_myGlobals); int mesh=FEM_Mesh_default_read(); // Get node data nnodes=FEM_Mesh_get_length(mesh,FEM_NODE); g.coord=new vector2d[nnodes]; FEM_Mesh_data(mesh, FEM_NODE, FEM_DATA+0, (double*)g.coord, 0, nnodes, FEM_DOUBLE, 2);

94. Charm Workshop 200594 // Read element data from framework nelems=FEM_Mesh_get_length(mesh,FEM_ELEM+0); g.nnodes=nnodes; g.nelems=nelems; g.conn=new connRec[nelems]; g.S11=new double[nelems]; g.S22=new double[nelems]; g.S12=new double[nelems]; FEM_Mesh_data(mesh, FEM_ELEM+0, FEM_CONN, (int *)g.conn, 0, nelems, FEM_INDEX_0, 3);

95. Charm Workshop 200595 //Initialize associated data g.R_net=new vector2d[nnodes]; //Net force g.d=new vector2d[nnodes];//Node displacement g.v=new vector2d[nnodes];//Node velocity g.a=new vector2d[nnodes];//Node acceleration for (i=0;i<nnodes;i++) g.R_net[i]=g.d[i]=g.v[i]=g.a[i]=vector2d(0.0); int fid=FEM_Create_simple_field(FEM_DOUBLE,2);

96. Charm Workshop 200596 int tSteps=5000; for (int t=0;t<tSteps;t++) { //Structural mechanics //Compute forces on nodes exerted by elements CST_NL(g.coord,g.conn,g.R_net,g.d,matConst,nnodes,nelems,g.S11, g.S22,g.S12); //Communicate net force on shared nodesFEM_Update_field(fid,g.R_net); //Advance node positions advanceNodes(dt,nnodes,g.coord,g.R_net,g.a,g.v,g.d,0); /* perform migration-based load balancing */ if (t%1024==0) FEM_Migrate();

97. Charm Workshop 200597 if (t%1024==0) { //Publish data for Visualization NetFEM n=NetFEM_Begin(FEM_My_partition(),t, 2,NetFEM_POINTAT); NetFEM_Nodes(n,nnodes,(double*)g.coord,"Position (m)"); NetFEM_Vector(n,(double*)g.d,"Displacement (m)"); NetFEM_Vector(n,(double*)g.v,"Velocity (m/s)"); NetFEM_Elements(n,nelems,3,(int*)g.conn,"Triangles"); NetFEM_Scalar(n,g.S11,1,"X Stress (pure)"); NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)"); NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)"); NetFEM_End(n); } } // end timeloop }// end driver

98. Conclusions

99. Charm Workshop 200599 Conclusions: ParFUM ParFUM is a functional, freely available Unstructured Mesh Framework. ParFUM has a wide range of useful features: Load Balancing, Fault Tolerance, … Parallel Partitioning Parallel Incremental Adaptivity, Remeshing, and Refinement Visualization Tools Collision/Contact Detection Library Matrix Solver

100. ParFUM Matrix Solver Library

101. Charm Workshop 2005101 IFEM: Intro to Matrix-based ParFUM Many ParFUM applications use direct Matrix-Free methods for solving. ParFUM does provide a matrix solver library.

102. Charm Workshop 2005102 IFEM: Intro to Matrix-based ParFUM Normal computations run element-by-element Only consider one element at a time Matrix-based computations are holistic One big matrix represents the whole system Solve the matrix == solve the problem Row of matrix solves for “Degree of Freedom” E.g., a node’s X coordinate Typically a row has only a few nonzero entries Around 10 6 degrees of freedom--10 6 rows

103. Charm Workshop 2005103 10 6 Rows is a Big Matrix Normal matrix operations are O(n 3 ) Can’t use Gaussian elimination, Jacobi, QR, … Almost everybody uses iterative solvers Based on a fast “guess-and-check” method Each iteration consists of a matrix-vector product Typically converges in 10 to 100 iterations Popular iterative solvers: Conjugate gradient (CG) Bidirectional Conjugate Gradient (BiCG) Generalized modified residual (GMRES)

104. Charm Workshop 2005104 Matrix Example: ZHooke’s spring law: ZF=-kx ZF1=-k(D1-D2) ZLinear relation between node force and node displacement ZElement corresponds to little 2x2 matrix F1 F2 = D1 D2 -k k k -k D1D2

105. Charm Workshop 2005105 Matrix Example: 2 Elements F1 F2 F3 = D1 D2 D3 -k k 0 k -k-k k 0 k -k D1D2 D3 AB

106. Charm Workshop 2005106 Matrix Example: Derivation F1 F2 F3 = D1 D2 D3 -k k k –k-k k k -k D1D2 D3 AB A B

107. Charm Workshop 2005107 Matrix Example: FEM D1D2 D3 AB D2 F1 F2A = D1 D2 -k k k -k F2B F3 = D2 D3 -k k k -k Shared Node Shared Node Later, sum partial forces across shared nodes to get: F2=F2A+F2B

108. Charm Workshop 2005108 IFEM: Bottom Line If you can do matrix-vector product, you can run iterative solvers like conjugate gradient Matrix-vector product amounts to the usual ParFUM calculation (displacements to forces), plus the usual ParFUM communication (update shared nodes) You don’t even need to store the matrix Users can easily solve iterative problems with the ParFUM framework and IFEM solvers!

109. The End Thanks