1. Ghost Elements

2. Ghost Elements: Overview Most FEM programs communicates via shared nodes, using FEM_Update_field Most FEM programs communicates via shared nodes, using FEM_Update_field Some computations require read-only copies of remote elements—“ghosts” Some computations require read-only copies of remote elements—“ghosts” Stencil-type finite volume computations Stencil-type finite volume computations The push form of matrix-vector product The push form of matrix-vector product Many kinds of mesh modification Many kinds of mesh modification Ghosts are a recent addition to the FEM framework Ghosts are a recent addition to the FEM framework

3. Ghosts: 2D Example 1234 12 Ghost of 3 Ghost of 2 34 Serial Mesh Left Chunk Right Chunk

4. Building Ghosts: Add ghost elements layer-by-layer from init Add ghost elements layer-by-layer from init A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes For 2D, a tuple might be a 2-node edge For 2D, a tuple might be a 2-node edge For 3D, a tuple might be a 4-node face For 3D, a tuple might be a 4-node face You specify a ghost layer with FEM_Add_ghost_layer(tupleSize,ghostNodes) You specify a ghost layer with FEM_Add_ghost_layer(tupleSize,ghostNodes) ghostNodes indicates whether to add ghost nodes as well as ghost elements. ghostNodes indicates whether to add ghost nodes as well as ghost elements.

5. Building Ghosts: FEM_Add_ghost_elem(e,t,elem2tuple) FEM_Add_ghost_elem(e,t,elem2tuple) e is the element type e is the element type t is the number of tuples per element t is the number of tuples per element elem2tuple maps an element to its tuples: elem2tuple maps an element to its tuples: A tupleSize by t array of integers A tupleSize by t array of integers Contains element-local node numbers Contains element-local node numbers Repeat this call for each ghost element type Repeat this call for each ghost element type

6. Ghosts: Node adjacency /* Node-adjacency: triangles have 3 nodes */ FEM_Add_ghost_layer(1,0); /* 1 node per tuple */ const static int tri2node[]={0,1,2}; FEM_Add_ghost_elem(0,3,tri2node); 0 1 2

7. Ghosts: Edge adjacency /* Edge-adjacency: triangles have 3 edges */ FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */ const static int tri2edge[]={0,1, 1,2, 2,0}; FEM_Add_ghost_elem(0,3,tri2edge); 0 1 2

8. Extracting and Using Ghosts Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end Ghosts are always given larger numbers than non-ghosts—that is, ghosts are at the end FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) FEM_Get_node_ghost() and FEM_Get_elem_ghost(e) Return the index of the first ghost node or element Return the index of the first ghost node or element FEM_Update_ghost_field(fid,e,data) FEM_Update_ghost_field(fid,e,data) Obtain other processor’s data (formatted like fid) for each ghost element of type e Obtain other processor’s data (formatted like fid) for each ghost element of type e 0eg

9. Ghost Elements: Sub-Mesh

10. Ghost Elements

11. Ghosts and Symmetries

12. In addition to cross-processor ghosts, can build ghosts to model problem symmetries In addition to cross-processor ghosts, can build ghosts to model problem symmetries Translational and rotational periodicities Translational and rotational periodicities Mirror symmetry Mirror symmetry FEM_Add_linear_periodicity(nFaces,nPer, facesA,facesB, nNodes,nodeLocs) FEM_Add_linear_periodicity(nFaces,nPer, facesA,facesB, nNodes,nodeLocs) Identify these two lists of faces under linear periodicity, and build ghosts to match Identify these two lists of faces under linear periodicity, and build ghosts to match

13. Symmetry Ghosts: 2D Example 1234 12 Ghost of 3 Ghost of 2 34 Serial Mesh Left ChunkRight Chunk Sym. Ghost 4 Sym. Ghost 1 Horizontal Periodicity

14. Symmetry Ghost Elements

15. Symmetry-Ghost Elements

16. Ghosts and Symmetries: Update

17. NetFEM Client

18. NetFEM Client: Pretty pictures of wave dispersion around a crack

19. NetFEM: Zoom in

20. NetFEM: Outline Elements

21. NetFEM: Point Nodes

22. NetFEM Server

23. NetFEM Server Side: Overview To allow the NetFEM client to connect, you add NetFEM registration calls to your server To allow the NetFEM client to connect, you add NetFEM registration calls to your server Register nodes and element types Register nodes and element types Register data items: scalars or spatial vectors associated with each node or element Register data items: scalars or spatial vectors associated with each node or element You provide the display name and units for each data item You provide the display name and units for each data item Link your program with “-module netfem” Link your program with “-module netfem” Run with “++server”, and connect! Run with “++server”, and connect!

24. NetFEM Server Side: Setup n=NetFEM_Begin(FEM_My_partition(),timestep, dim,NetFEM_POINTAT) n=NetFEM_Begin(FEM_My_partition(),timestep, dim,NetFEM_POINTAT) Call this each time through your timeloop; or skip Call this each time through your timeloop; or skip timestep identifies this data update timestep identifies this data update dim is the spatial dimension—must be 2 or 3 dim is the spatial dimension—must be 2 or 3 Returns a NetFEM handle n used by everything else Returns a NetFEM handle n used by everything else NetFEM_End(n) NetFEM_End(n) Finishes update n Finishes update n

25. NetFEM Server Side: Nodes NetFEM_Nodes(n,nnodes,coord,”Position (m)”) NetFEM_Nodes(n,nnodes,coord,”Position (m)”) Registers node locations with NetFEM—future vectors and scalars will be associated with nodes Registers node locations with NetFEM—future vectors and scalars will be associated with nodes n is the handle returned by NetFEM_Begin n is the handle returned by NetFEM_Begin nnodes is the number of nodes nnodes is the number of nodes coord is a dim by nnodes array of doubles coord is a dim by nnodes array of doubles The string describes the coordinate system and meaning of nodes The string describes the coordinate system and meaning of nodes Currently, there can only be one call to nodes Currently, there can only be one call to nodes

26. NetFEM: Node Displacement

27. NetFEM Server Side: Elements NetFEM_Elements(n,nelem,nodeper, conn,”Triangles”) NetFEM_Elements(n,nelem,nodeper, conn,”Triangles”) Registers elements with NetFEM—future vectors and scalars will be associated with these elements Registers elements with NetFEM—future vectors and scalars will be associated with these elements n is the handle returned by NetFEM_Begin n is the handle returned by NetFEM_Begin nelem is the number of elements nelem is the number of elements nodeper is the number of nodes per element nodeper is the number of nodes per element conn is a nodeper by nelem array of node indices conn is a nodeper by nelem array of node indices The string describes the kind of element The string describes the kind of element Repeat to register several kinds of element Repeat to register several kinds of element Perhaps: Triangles, squares, pentagons, … Perhaps: Triangles, squares, pentagons, …

28. NetFEM: Element Stress

29. NetFEM Server Side: Vectors NetFEM_Vector(n,val,”Displacement (m)”) NetFEM_Vector(n,val,”Displacement (m)”) Registers a spatial vector with each node or element Registers a spatial vector with each node or element Whichever kind was registered last Whichever kind was registered last n is the handle returned by NetFEM_Begin n is the handle returned by NetFEM_Begin val is a dim by nitems array of doubles val is a dim by nitems array of doubles There’s also a more general NetFEM_Vector_field in the manual There’s also a more general NetFEM_Vector_field in the manual The string describes the meaning and units of the vectors The string describes the meaning and units of the vectors Repeat to register multiple sets of vectors Repeat to register multiple sets of vectors Perhaps: Displacement, velocity, acceleration, rotation, … Perhaps: Displacement, velocity, acceleration, rotation, …

30. NetFEM: Element Velocity

31. NetFEM Server Side: Scalars NetFEM_Scalar(n,val,s,”Displacement (m)”) NetFEM_Scalar(n,val,s,”Displacement (m)”) Registers s scalars with each node or element Registers s scalars with each node or element Whichever kind was registered last Whichever kind was registered last n is the handle returned by NetFEM_Begin n is the handle returned by NetFEM_Begin val is an s by nitems array of doubles val is an s by nitems array of doubles There’s also a more general NetFEM_Scalar_field in the manual There’s also a more general NetFEM_Scalar_field in the manual s is the number of doubles for each node or element s is the number of doubles for each node or element The string describes the meaning and units of the scalars The string describes the meaning and units of the scalars Repeat to register multiple sets of scalars Repeat to register multiple sets of scalars Perhaps: Stress, plasticity, node type, damage, … Perhaps: Stress, plasticity, node type, damage, …

32. NetFEM Server Side: 2D Example integer :: t,n, numnp, numel real*8, dimension(2,numnp) :: coor,d,v,a integer, dimension(3,numel) :: conn n=NetFEM_Begin(FEM_My_partition(),t,2,NetFEM_POINTAT) CALL NetFEM_Nodes(n,numnp,coor,'Position (m)') CALL NetFEM_Vector(n,d,'Displacement (m)') CALL NetFEM_Vector(n,v,'Velocity (m/s)') CALL NetFEM_Vector(n,a,'Acceleration (m/s^2)') CALL NetFEM_Elements(n,numel,3,conn,'Triangles') CALL NetFEM_Scalar(n,stress,1,'Stress (pure)') CALL NetFEM_End(n)

33. NetFEM: Conclusion Easy, general way to get output from an FEM computation Easy, general way to get output from an FEM computation Client configures itself based on server Client configures itself based on server Client can be run anywhere (from home!) Client can be run anywhere (from home!) Server performance impact minimal (1  s!) Server performance impact minimal (1  s!) Future work: Future work: Support multiple chunks per processor Support multiple chunks per processor Non-network, file-based version Non-network, file-based version Movie mode Movie mode

34. Multiple Modules

35. Use of 2 or more CHARM++ frameworks in the same program Use of 2 or more CHARM++ frameworks in the same program FEM—multiple unstructured mesh chunks FEM—multiple unstructured mesh chunks MBLOCK—multiple structured mesh blocks MBLOCK—multiple structured mesh blocks AMPI—Adaptive MPI-on-Charm++ AMPI—Adaptive MPI-on-Charm++ All based on the Threaded CHARM++ framework (TCHARM) All based on the Threaded CHARM++ framework (TCHARM) For example, we may want to use AMPI in our FEM program for exchanging information between FEM chunks For example, we may want to use AMPI in our FEM program for exchanging information between FEM chunks

36. Details Can compose FEM programs with other modules by just calling that module’s attach routine from init() Can compose FEM programs with other modules by just calling that module’s attach routine from init() For example: For example: void init(void) { //Start AMPI, to allow drivers to use MPI calls: //Start AMPI, to allow drivers to use MPI calls:MPI_Attach(“myAMPIFEM”); //.. Use FEM_Set() calls as usual.. }

37. Example #include ‘fem.h’ #include ‘mpi.h’ void driver(void) { //..use FEM_Get calls as usual.. //..use FEM_Get calls as usual.. //Broadcast “data” from chunk 0: //Broadcast “data” from chunk 0: MPI_Bcast(&data,1,MPI_DOUBLE,0,MPI_COMM_WORLD); MPI_Bcast(&data,1,MPI_DOUBLE,0,MPI_COMM_WORLD); //...timeloop: Use FEM_Update_field calls as usual... //...timeloop: Use FEM_Update_field calls as usual... if (dataToSend) if (dataToSend) MPI_Send(&data,4,MPI_INT,dest,tag,MPI_COMM_WORLD); MPI_Send(&data,4,MPI_INT,dest,tag,MPI_COMM_WORLD); else else MPI_Recv(&data,4,MPI_INT,src,tag,MPI_COMM_WORLD,&status); MPI_Recv(&data,4,MPI_INT,src,tag,MPI_COMM_WORLD,&status);}

38. Multiple Modules: Conclusion Easy to use other modules from FEM framework Easy to use other modules from FEM framework Just call MPI_Attach from init, and link with “-module ampi” Just call MPI_Attach from init, and link with “-module ampi” We could also have specified how to combine frameworks by writing a special startup routine named TCHARM_User_setup() We could also have specified how to combine frameworks by writing a special startup routine named TCHARM_User_setup() Not FEM-centric: overrides the normal call to init() Not FEM-centric: overrides the normal call to init() Allows you to call main computation routine something other than driver() Allows you to call main computation routine something other than driver() See TCHARM manual for details See TCHARM manual for details