1 FEM Framework Tutorial Sayantan Chakravorty 10/19/2004

2 Roadmap Why use FEM? FEM Concepts FEM Program Structure FEM Basic Calls FEM Advanced Calls Extra Features

3 Why use FEM?

4 Why use the FEM Framework? Makes parallelizing a serial code faster and easier Handles mesh partitioning Handles communication Handles load balancing (via Charm) Allows extra features NetFEM Visualizer Collision Detection Library Parallel Refinement Library IFEM Matrix Library

5 Why not use the FEM Framework? Does not help you write serial code But it does help with parallel Another thing to learn But it has a manual and examples Another thing to break But it runs on native MPI as well as AMPI

6 Fem Framework Users CSAR Rocflu: Fluids solver CPSD SpaceTime meshing Frac3D Fracture Mechanics Dendritic Growth Metal Solidification process

7 FEM Concepts

8 FEM Basics FEM programs manipulate elements and nodes Element is a portion of problem domain, surrounded by nodes Node is one point in the domain

9 Serial FEM Mesh N5N4N2E3 N4N2N1E2 N4N3N1E1 Surrounding Nodes Element

10 Partitioned Mesh N3N2N1E2 N4N3N1E1 Surrounding Nodes Element N3N2N1E1 Surrounding Nodes Element Shared Nodes N3N4 N1N2 BA

11 FEM Parallel Model: Shared Nodes “Shared Node” model Element computations based on values of surrounding nodes Node values are sum of surrounding elements Example: Mechanical Simulation Element stresses are computed from locations of element’s nodes Sum up forces at nodes

12 FEM Mesh: Node Communication Summing forces from other processors only takes one call: FEM_Update_field Adds values from shared nodes

13 FEM Parallel Model: Ghosts “Ghost” model Element computations based only on values of surrounding nodes and elements Example: Fluid Dynamics Element pressures and velocities come from neighboring element pressures and velocities, and node locations

14 FEM Mesh: Ghosts Ghost of 3 Ghost of 2 34 Serial Mesh Left Chunk Right Chunk Partitioning Communication

15 FEM Program Structure

16 Serial FEM Program Structure read mesh, connectivity, boundary conditions time loop element loop- Element deformation applies forces to surrounding nodes node loop- Forces and boundary conditions change node positions end time loop write out mesh data for postprocessing

17 Parallelization Partition the FEM Mesh into multiple chunks Distribute elements, replicate shared nodes and/or add ghosts Keep track of communication Partition so that communication is minimized

18 Parallel FEM Program read/get chunk mesh data, connectivity, shared nodes chunk time loop element loop- Element deformation applies forces to surrounding nodes node loop- Forces and boundary conditions change node positions end time loop write chunk mesh data for postprocessing

19 FEM Framework Program Consists of at least two user- written subroutines init driver init is called on chunk 0 driver is called on every chunk

20 init subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine

21 driver subroutine driver get local mesh chunk time loop FEM computations communication more FEM computations end time loop end subroutine

22 Structure of an FEM Application init() Update driver

23 FEM Mesh Access Calls

24 void FEM_Mesh_data( int mesh, int entity, int attr, void *data, int first, int length, int datatype, int width ); void FEM_Mesh_data( int mesh, int entity, int attr, void *data, int first, int length, int datatype, int width ); Get/Set, and multi-mesh support NODE, ELEM, SPARSE (+GHOST) DATA, CONN, SYM, GLOBALNO,… Apply to first…first+length-1 User data: width x length array User data formatting

25 FEM_Mesh_data( mesh, FEM_NODE, FEM_DATA+23, coord, 0,nNodes, FEM_DOUBLE,3 ); Mesh Access: Example e.g., FEM_Mesh_default_read() We’re changing node values User data (tag 23) Change all the nodes An array of 3 x nNodes doubles 3 doubles/node

26 FEM Communication Calls

27 Node Fields Framework handles combining data for shared nodes and keeps them in sync Framework does not understand meaning of node fields, only their location and types Framework needs to be informed of locations and types of fields Create_field once, Update_field every timestep

28 Create a Field integer function FEM_Create_simple_field( datatype, len) integer, intent(in) :: datatype, len

29 Update Field: Shared Nodes subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes

30 FEM Ghost Layers

31 Ghost Elements: Overview Most FEM programs communicate via shared nodes Some computations require read-only copies of remote elements—“ghosts” Stencil-type finite volume computation Many kinds of mesh modification

32 Ghosts: 2D Example Ghost of 3 Ghost of 2 34 Serial Mesh Left Chunk Right Chunk

33 Building Ghosts: Add ghost elements layer-by-layer from init A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes For 2D, a tuple might be a 2-node edge For 3D, a tuple might be a 4-node face You specify a ghost layer with FEM_Add_ghost_layer(tupleSize,ghostNodes) ghostNodes indicates whether to add ghost nodes as well as ghost elements.

34 Building Ghosts: FEM_Add_ghost_elem(e,t,elem2tuple) e is the element type t is the number of tuples per element elem2tuple maps an element to its tuples: A tupleSize by t array of integers Contains element-local node numbers Repeat this call for each ghost element type

35 Ghosts: Node adjacency /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1,0); /* 1 node per tuple */ const static int tri2node[]={0,1,2}; FEM_Add_ghost_elem(0,3,tri2node); 0 1 2

36 Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */ const static int tri2edge[]={0,1, 1,2, 2,0}; FEM_Add_ghost_elem(0,3,tri2edge); 0 1 2

37 Extracting and Using Ghosts Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) Return the index of the first ghost node or element FEM_Update_ghost_field(fid,e,data) Obtain other processor’s data (formatted like fid) for each ghost element of type e 0eg

38 Update Field: Ghosts subroutine FEM_Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts

39 Ghost Example: Mesh

40 Ghost Example: Ghost Elements

41 FEM Installation

42 Where to Get It ? FEM Framework is included in Charm++ distribution, available under CVS CSH: setenv CVSROOT Or BASH: export You should now be able to do a > cvs login (no password needed, just type [Enter] at prompt) and then > cvs co -P charm to get the entire Charm++ source: FEM is in charm/src/libs/ck-libs

43 How to Build It ? > cd charm and do >./build FEM net-linux -O This will make a net-linux directory, with bin, include, lib etc subdirectories. Platforms: net-sol, mpi-origin, mpi-linux etc.

44 How to Compile & Link ? Use “charmc”: available under bin a multi-lingual compiler driver, understands f90 Knows where modules and libraries are Portable across machines and compilers Linking use “-language femf” : for F90 Use “–language fem” : for C/C++ See example Makefiles charm/examples/fem/…

45 How to Run ? Charmrun A portable parallel job execution script Specify number of processors: +pN Specify number of chunks: +vpN Special “nodelist” file for net-* versions

46 Example Nodelist File: $(HOME)/.nodelist group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.edu etc…./charmrun pgm +p4

47 Advanced FEM Calls

48 Advanced: FEM Migration

49 Advanced API: Migration Chunks may not be computationally equal: Results in load imbalance Multiple chunks per processor Chunks cannot have writable global data Automatic load balancing Migrate chunks to balance load How to migrate allocated data for chunks ? Embed it in a user-defined type

50 Chunk Data Example MODULE my_block_mod TYPE my_block INTEGER :: n1,n2x,n2y REAL*8, POINTER, DIMENSION(:,:) :: arr END TYPE END MODULE

51 Pack/Unpack (PUP) Routine SUBROUTINE pup_my_block(p,m) USE my_block_mod USE pupmod INTEGER :: p TYPE(my_block) :: m call fpup_int(p,m%n1) call fpup_int(p,m%n2x) call fpup_int(p,m%n2y) IF (fpup_isUnpacking(p)) THEN ALLOCATE(m%arr(m%n2x,m%n2y)) END IF call fpup_doubles(p,m%arr,m%n2x*m%n2y) IF (fpup_isDeleting(p)) THEN DEALLOCATE(m%arr) END IF END SUBROUTINE

52 Registering Chunk Data !- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p,m) use my_block_mod INTEGER :: p TYPE(my_block) :: m end subroutine end interface TYPE(my_block) :: m CALL FEM_Register(m,pup_my_block)

53 Migration Every chunk driver calls FEM_Migrate Framework calls PUP for getting the size of packed data For packing data Chunk migrates to new processor Framework calls PUP for unpacking Driver returns from FEM_Migrate

54 Advanced: Complicated Fields

55 Node Fields

56 FEM_Create_Field function integer :: FEM_Create_Field( base_type, vec_len, offset, dist) integer, intent(in) :: base_type, vec_len, offset, dist Base_type FEM_BYTE- INTEGER*1, or CHARACTER*1 FEM_INT- INTEGER*4 FEM_REAL- REAL*4 FEM_DOUBLE- DOUBLE PRECISION, or REAL*8

57 Create_field Example ! 3D Force for each node ! stored as 3*n real*8 array REAL*8 ALLOCATABLE, DIMENTION(:) :: nodeForce INTEGER :: fid... allocate nodeForce as 3*n_nodes... fid = FEM_Create_Field(FEM_DOUBLE,3, foffsetof(nodeForce(1),nodeForce(1)), foffsetof(nodeForce(1),nodeForce(4))

58 Create_field Example ! 3D force is contained as fXYZ variable ! in a user-defined type node_type TYPE(node_type), ALLOCATABLE, DIMENTION(:) :: nodes INTEGER :: fid...allocate nodes array as n_nodes... fid = FEM_Create_Field(FEM_DOUBLE,3, foffsetof(nodes(1), nodes(1)%fXYZ), foffsetof(nodes(1), nodes(2)) )

59 Advanced: FEM on MPI

60 FEM on MPI FEM routines perform their communication using MPI calls This means you can call FEM routines from an MPI program; or call MPI routines from an FEM program

61 Chunk Data Example PROGRAM main include ‘femf.h’ include ‘mpif.h’ CALL MPI_Init() CALL FEM_Init(MPI_COMM_WORLD)... MPI or FEM routines... END PROGRAM

62 Extra FEM Features

63 Collision Detection

64 Charm++ Collision Detection Detect collisions (intersections) between objects scattered across processors Built on Charm++ Arrays Overlay regular 3D sparse grid of voxels (boxes) Send objects to all voxels they touch Collect collisions from each voxel Collision response is left to caller

65 Mesh Adaptation

66 Parallel Mesh Refinement 2D refinement 2D coarsening 3D refinement Integration with FEM

67 Triangular Mesh Refinement To refine, split the longest edge: But if split neighbor has a longer edge, split his edge first Refinement propagates across mesh, but preserves mesh quality Initial 2D parallel implementation built on Charm++ Integrated with FEM Framework FEM_Refine2D_Newmesh FEM_Refine2D_Split Input: Nodes, coordinates, elements and desired areas for elements Optional: Boundary values for edges Effect: Updates FEM mesh, interpolates user data and sets boundary values for new nodes

68 TMR : Results Initial Mesh: coarse Refined Mesh: later in the simulation

69 2D coarsening Collapse the smallest edge of an element Makes the adjacent elements larger Being integrated into FEM

70 Parallel 3D refinement Longest Edge based refinement Longest Face based refinement Internal refinement

71 NetFEM

72 NetFEM Client: pretty pictures Wave dispersion off a crack (simplified frac3d)

73 NetFEM: Easy Visualization Can interact with a running FEM application Uses Charm++ CCS (Client-server) library Easy to “publish” attributes NetFEM n=NetFEM_Begin(2,t,NetFEM_POINTAT); NetFEM_Nodes(n,nnodes,(double *)g.coord,"Position (m)"); NetFEM_Vector(n,(double *)g.d,"Displacement (m)"); NetFEM_Vector(n,(double *)g.v,"Velocity (m/s)"); NetFEM_Elements(n,nelems,3,(int *)g.conn,"Triangles"); NetFEM_Scalar(n,g.S11,1,"X Stress (pure)"); NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)"); NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)"); NetFEM_End(n); Author: Orion Lawlor

74 NetFEM: Easy visualization

75 NetFEM: Zoom in

76 NetFEM: Outline Elements

77 NetFEM: Point Nodes

78 NetFEM Server Side: Overview To allow the NetFEM client to connect, you add NetFEM registration calls to your server Register nodes and element types Register data items: scalars or spatial vectors associated with each node or element You provide the display name and units for each data item Link your program with “-module netfem” Run with “++server”, and connect!

79 NetFEM Server Side: Setup n=NetFEM_Begin(FEM_My_partition(),ti mestep, dim,NetFEM_POINTAT) Call this each time through your timeloop; or skip timestep identifies this data update dim is the spatial dimension—must be 2 or 3 Returns a NetFEM handle n used by everything else NetFEM_End(n) Finishes update n

80 NetFEM Server Side: Nodes NetFEM_Nodes(n,nnodes,coord,”Posi tion (m)”) Registers node locations with NetFEM—future vectors and scalars will be associated with nodes n is the handle returned by NetFEM_Begin nnodes is the number of nodes coord is a dim by nnodes array of doubles The string describes the coordinate system and meaning of nodes Currently, there can only be one call to nodes

81 NetFEM: Node Displacement

82 NetFEM Server Side: Elements NetFEM_Elements(n,nelem,nodeper, conn,”Triangles”) Registers elements with NetFEM—future vectors and scalars will be associated with these elements n is the handle returned by NetFEM_Begin nelem is the number of elements nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices The string describes the kind of element Repeat to register several kinds of element Perhaps: Triangles, squares, pentagons, …

83 NetFEM: Element Stress

84 NetFEM Server Side: Vectors NetFEM_Vector(n,val,”Displacement (m)”) Registers a spatial vector with each node or element Whichever kind was registered last n is the handle returned by NetFEM_Begin val is a dim by nitems array of doubles There’s also a more general NetFEM_Vector_field in the manual The string describes the meaning and units of the vectors Repeat to register multiple sets of vectors Perhaps: Displacement, velocity, acceleration, rotation, …

85 NetFEM: Element Velocity

86 NetFEM Server Side: Scalars NetFEM_Scalar(n,val,s,”Displacement (m)”) Registers s scalars with each node or element Whichever kind was registered last n is the handle returned by NetFEM_Begin val is an s by nitems array of doubles There’s also a more general NetFEM_Scalar_field in the manual s is the number of doubles for each node or element The string describes the meaning and units of the scalars Repeat to register multiple sets of scalars Perhaps: Stress, plasticity, node type, damage, …

87 NetFEM Server Side: 2D Example integer :: t,n, numnp, numel real*8, dimension(2,numnp) :: coor,d,v,a integer, dimension(3,numel) :: conn n=NetFEM_Begin(FEM_My_partition(),t,2,NetFEM_POINTAT) CALL NetFEM_Nodes(n,numnp,coor,'Position (m)') CALL NetFEM_Vector(n,d,'Displacement (m)') CALL NetFEM_Vector(n,v,'Velocity (m/s)') CALL NetFEM_Vector(n,a,'Acceleration (m/s^2)') CALL NetFEM_Elements(n,numel,3,conn,'Triangles') CALL NetFEM_Scalar(n,stress,1,'Stress (pure)') CALL NetFEM_End(n)

88 NetFEM: Conclusion Easy, general way to get output from an FEM computation Client configures itself based on server Client can be run anywhere (from home!) Server performance impact minimal (1s!) Future work: Support multiple chunks per processor Movie mode

89 Conclusions: Charm++ FEM Easy to parallelize existing codes Flexible: shared nodes or ghosts High performance Extra features Visualization with NetFEM Collision Matrix methods with IFEM

90 IFEM: Intro to Matrix-based FEM Normal computations run element- by-element Only consider one element at a time Matrix-based computations are holistic One big matrix represents the whole system Solve the matrix == solve the problem Row of matrix solves for “Degree of Freedom” E.g., a node’s X coordinate Typically a row has only a few nonzero entries Around 10 6 degrees of freedom rows

Rows is a Big Matrix Normal matrix operations are O(n 3 ) Can’t use Gaussian elimination, Jacobi, QR, … Almost everybody uses iterative solvers Based on a fast “guess-and-check” method Each iteration consists of a matrix-vector product Typically converges in 10 to 100 iterations Popular iterative solvers: Conjugate gradient (CG) Bidirectional Conjugate Gradient (BiCG) Generalized modified residual (GMRES)

92 ILSI: Iterative Linear Solver Interface Provides solvers with operations they need: Parallel matrix-vector product (written by user) Parallel dot-product class ILSI_Comm { virtual void matrixVectorProduct(const double *src,double *dest) =0; virtual double dotProduct(const double *a,const double *b) =0; }; Iterative solvers use these operations to solve the matrix Writing iterative solvers: an active research area Krylov Subspaces, Residuals, Quadratic forms…

93 Matrix Example: Hooke’s spring law: F=-kx F1=-k(D1-D2) Linear relation between node force and node displacement Element corresponds to little 2x2 matrix F1 F2 = D1 D2 -k k k -k D1D2

94 Matrix Example: 2 Elements F1 F2 F3 = D1 D2 D3 -k k 0 k -k-k k 0 k -k D1D2 D3 AB

95 Matrix Example: Derivation F1 F2 F3 = D1 D2 D3 -k k k –k-k k k -k D1D2 D3 AB A B

96 Matrix Example: FEM D1D2 D3 AB D2 F1 F2A = D1 D2 -k k k -k F2B F3 = D2 D3 -k k k -k Shared Node Shared Node Later, sum partial forces across shared nodes to get: F2=F2A+F2B

97 IFEM: Bottom Line If you can do matrix-vector product, you can run iterative solvers like conjugate gradient Matrix-vector product amounts to the usual FEM calculation (displacements to forces), plus the usual FEM communication (update shared nodes) You don’t even need to store the matrix Users can easily solve iterative problems with the FEM framework and IFEM solvers!

98 Additional features IFEM: Iterative FEM Linear Solver Interface FEM supports Symmetries Provides support for data transfer between meshes