1. Basic Concepts of FEM Framework & API
Gunavardhan Kakulapati

2. Introduction
FEM Framework overview
Using the framework API
Installation
Conversion Example

3. Motivation Finite Element Method Do it in parallel !! Used extensively
Computationally intensive
Do it in parallel !!

4. FEM Programs FEM programs manipulate elements and nodes
Element is a portion of problem domain, surrounded by nodes
Element computation: based on fields of surrounding nodes
Elements contribute to fields of surrounding nodes

5. FEM Mesh
Element
Surrounding Nodes E1 N1 N3 N4 E2 N2 E3 N5

6. FEM Program Structure read mesh, connectivity, boundary conditions
time loop           element loop- Element deformation applies
forces to surrounding nodes           node loop- Forces and boundary conditions
change node positions end time loop
write out mesh data for postprocessing

7. Parallelization Partition the FEM Mesh into multiple chunks
Distribute elements, replicate shared nodes
Shared nodes = Communication !!
Partition so that communication is minimized

8. Partitioning Element Surrounding Nodes E1 N1 N3 N4 E2 N2 Element
Shared Nodes A B N2 N1 N4 N3

9. Parallel FEM Program
read/get chunk mesh data, connectivity, shared nodes
chunk time loop           element loop- Element deformation applies
forces to surrounding nodes           <update forces on shared nodes>           node loop- Forces and boundary conditions
change node positions end time loop
write chunk mesh data for postprocessing
>>

10. FEM Framework: Goals
Separate parallel implementation from numerical algorithms
Parallel version to closely resemble the serial program
Allow features like load balancing, visualization.

11. FEM Framework: Responsibilities
FEM Application
(Initialize, Registration of Nodal Attributes, Loops Over Elements, Finalize)
FEM Framework
(Update of Nodal properties, Reductions over nodes or partitions)
Partitioner
Combiner
METIS
Charm++
(Dynamic Load Balancing, Communication)
I/O

12. FEM Framework Program
May contain user-written, library-called subroutines:
init
driver
mesh_updated (more on this later…)
init and mesh_updated are called on processor 0
driver is called on every chunk

13. Structure of an FEM Application
init()
Update
driver
driver
Update
driver
Update
Shared Nodes
Shared Nodes

14. init subroutine init read the serial mesh and configuration data
inform the framework about
the mesh end subroutine

15. driver
subroutine driver           get local mesh chunk           time loop                FEM computations                update shared node fields                more FEM computations           end time loop end subroutine

16. Data output Parallel output at the end of driver
Possible to visualize the data rather than printing it out (netfem).
Framework calls like FEM_Mesh_updated. (more on this later …)
>>

17. Framework Calls FEM_Set_* FEM_Create_field FEM_Update_field
Called from initialization to set the serial mesh
Framework partitions mesh into chunks
FEM_Create_field
Registers a node data field with the framework, supports user data types
FEM_Update_field
Updates node data field across all processors
Handles all parallel communication
Other parallel calls (Reductions, etc.)

18. Framework Calls: Mesh FEM_Set_* FEM_Get_*
From init, these routines describe the mesh to the FEM Framework
Framework calls the partitioner
FEM_Get_*
From each chunk, the local portion of the mesh is obtained by the driver

19. FEM_*_elem FEM_Set_elem(int elType,int nEl,
int doublePerEl,int nodePerEl);
subroutine FEM_Set_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType,nEl,doublePerEl,nodePerEl
FEM_Get_elem(int elType,int* nEl,
int* doublePerEl,int* nodePerEl);
subroutine FEM_Get_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType integer, intent(out) :: nEl,doublePerEl,nodePerEl

20. FEM_*_node
subroutine FEM_Set_node(nNode,doublePerNode) integer, intent(in) :: nNode,doublePerNode
subroutine FEM_Get_node(nNode,doublePerNode) integer, intent(out) :: nNode,doublePerNode

21. Element Connectivity
subroutine FEM_Set_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nodePerEl,nEl) :: conn
subroutine FEM_Get_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nodePerEl,nEl) :: conn
subroutine FEM_Set_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nEl,nodePerEl) :: conn
subroutine FEM_Get_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nEl,nodePerEl) :: conn

22. Additional Data for Nodes and Elements
subroutine FEM_Set_node_data_r(data) REAL*8, intent(in),
dimension(doublePerNode,nNode) :: data
subroutine FEM_Get_node_data_r(data) REAL*8, intent(out),
subroutine FEM_Set_elem_data_r(data) REAL*8, intent(in),
dimension(doublePerElem,nElem) :: data
subroutine FEM_Get_elem_data_r(data) REAL*8, intent(out),

23. Node Fields
Framework handles combining data for shared nodes and keeps them in sync
Framework does not understand meaning of node fields, only their location and types
Framework needs to be informed of locations and types of fields
Create_field once, Update_field every timestep

24. FEM_Create_field To handle the updating of shared node values.
Tell the framework where the shared data items of each node are.
Creates a “field” and pass the field ID for updating shared nodal values.

25. FEM_Create_simple_field
function integer :: FEM_Create_simple_field(
base_type,
vec_len) integer, intent(in) :: base_type, vec_len
Base_type
FEM_BYTE- INTEGER*1, or CHARACTER*1
FEM_INT- INTEGER*4
FEM_REAL- REAL*4
FEM_DOUBLE- DOUBLE PRECISION, or REAL*8

26. Create_simple_field Example
! 3D Force for each node
! stored as 3*n real*8 array
REAL*8 ALLOCATABLE, DIMENSION(:) :: nodeForce INTEGER :: fid
... allocate nodeForce as 3*n_nodes...
fid = FEM_Create_simple_field(FEM_DOUBLE,3)

27. FEM_Create_field
function integer :: FEM_Create_Field(base_type, vec_len,
offset, dist)
integer, intent(in) :: base_type, vec_len, offset, dist

28. Node Fields

29. Create_field Example ! 3D force is contained as fXYZ variable
! in a user-defined type node_type
TYPE(node_type), ALLOCATABLE, DIMENSION(:) :: nodes INTEGER :: fid
...allocate nodes array as n_nodes...
fid = FEM_Create_Field(FEM_DOUBLE,3,          offsetof(nodes(1), nodes(1)%fXYZ),          offsetof(nodes(1), nodes(2)) )

30. Update and Reduce Field
subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes
subroutine FEM_Reduce_Field(fid,nodes,outVal,op) integer, intent(in) :: fid,op varies, intent(in) :: nodes varies, intent(out) :: outVal
op is
FEM_SUM
FEM_MIN
FEM_MAX

31. Utility function integer :: FEM_Num_Partitions()
function integer :: FEM_My_Partition()
function double precision :: FEM_Timer()
subroutine FEM_Print_Partition()
subroutine FEM_Print(str) character*, intent(in) :: str

32. Advanced FEM calls FEM_Update_mesh FEM_Add_node
Reassembles chunks of the mesh
FEM_Add_node
Adds a new node into the mesh

33. FEM_Update_mesh Reassembles all the chunks
Can be called only from driver
Must be called from all chunks
Useful scenarios
Giving out data as the simulation runs
Repartition the mesh
Serial output when simulation finishes

34. FEM_Update_mesh C call: Fortran call:
void FEM_Update_mesh(int callMeshUpdated, int doWhat)
Fortran call:
subroutine FEM_Update_mesh(callMeshUpdated,doWhat)
integer, intent(in) :: callMeshUpdated, doWhat

35. FEM_Update_mesh parameters
callMeshUpdated is non-zero => call mesh_updated (callMeshUpdated)
doWhat:
doWhat
Repartition
FEM_Update_mesh No
Non-blocking mesh_updated 1
Yes
Blocks for repartitioning 2
Blocking mesh_updated

36. FEM_Update_mesh examples
FEM_Update_mesh(k,0)
Call mesh_updated(k) on assembled mesh, while the driver continues
FEM_Update_mesh(k,1)
Repartition after mesh_updated
FEM_Update_mesh(k,2)
Block driver routines till mesh_updated(k)

37. Adding Nodes One can add new nodes, and update connectivity in driver
Use FEM_Set_* subroutines
New nodes are considered private
Framework can repartition the mesh
Optionally calls user’s mesh_updated subroutine

38. FEM_Add_node >> C call: Fortran call:
void FEM_Add_node(int localIdx,
int nBetween,
int * betweenNodes)
Fortran call:
subroutine FEM_Add_node(localIdx,nBetween,betweenNodes)
integer, intent(in) :: localIdx,nBetween
integer, intent(in) :: betweenNodes(nBetween)
>>

39. Installing FEM Framework

40. Where to Get It ? FEM Framework is included in Charm++ distribution,
available under CVS
CSH:
setenv CVSROOT
Or BASH:
export
You should now be able to do a
> cvs login (no password needed, just type [Enter] at prompt)
and then
> cvs co -P charm
to get the entire Charm++ source.

41. How to Build It ? > cd charm and do > ./build FEM net-linux -O
This will make a net-linux directory,
with bin, include, lib etc subdirectories.
Platforms: net-sol, mpi-origin, mpi-linux etc.

42. How to Write Programs ? Write from scratch Convert existing program
Concepts and API discussed earlier
Read the FEM-Framework Manual
Convert existing program
To be covered in next section

43. How to Compile & Link ? Use “charmc”: available under bin Linking
a multi-lingual compiler driver, understands f90
Knows where modules and libraries are
Portable across machines and compilers
Linking
use “-language femf” : for F90
Use “–language fem” : for C/C++
See example Makefiles
pgms/charm++/fem/…

44. How to Run ? Just run it! (net- versions only) Use Charmrun
Serial, but nice for debugging/testing
Use Charmrun
A portable parallel job execution script
Specify number of processors: +pN
Special “nodelist” file for net-* versions
Multiple chunks per processor: use +vpM

45. Charmrun Example ./charmrun pgm +p4 +vp8 >>
Nodelist File: $(HOME)/.nodelist group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.edu etc…
>>
./charmrun pgm +p4 +vp8

46. FEM Framework Conversion example

47. A Serial Program
Processing
Input
Output

48. A Parallel Framework Program
Processing
Input
Output
Parallel
Infrastructure

49. Real Names of Pieces Driver Init Finalize or Mesh_updated Charm++ FEM
Framework

50. Serial Example Program
F90 example
Reads input mesh in “Triangle” format
Does simple explicit mechanics computation (CST triangles)
Writes Tecplot output
Not a toy example (by Philippe Geubelle)
Reads a parameter file
Applies boundary conditions
Uses real mechanics

51. FEM Framework Version Uses all the same code as serial version
Main program becomes init subroutine
Split up init and driver
Copy entire mesh into framework (FEM_Set)
Copy portion of mesh out (FEM_Get)
Changes to time loop are minimal
Output poses an interesting problem

52. Splitting the program Declare variables call readTriGlobals Init
Read mesh conn. Data
Dynamics loop
Calc forces
Calc accl,vel
Apply bound. conditions Driver
Update displacements
Output current data

53. Init – Basic differences
Serial version
Declare variables
. . .
call readTriGlobals
Read mesh conn. data
Parallel version
Declare variables
. . .
call readTriGlobals
Read mesh conn. data
Call the set methods

54. Driver Parallel version Serial version Call get methods Dynamics loop
Create field
Dynamics loop
Calc forces
Update field
Calc accl,vel
Apply bound. Cond
Update disp
Visualize current data
Serial version
Dynamics loop
Calc forces
Calc accl,vel
Apply bound. Cond
Update disp
Output current data

55. More information:
Read the manual and FEM framework details at