1. 2D AFEAPI Overview Goals, Design Space Filling Curves Code Structure
Key generation/indexing scheme, mesh partitioning, ordering, hashtable
Code Structure
Preprocessing, AFEAPI code, postprocessing
Node, element, hashtable
Available at:

2. Difficulties of Parallel Adaptive Codes
Dynamic allocation of memory
Dynamic creation/deletion of objects (eg. nodes, elements)
Dynamic load-balancing
Dealing with on processor, off-processor and global information
Global constraints

3. Infrastructure Requirements
Ability to insert and delete objects during simulation
dynamic allocation and de-allocation of memory
Automatically distribute/redistribute data and computation
among processors Dynamic Load Balancing
Maintain irregularity and other refinement constraints on distributed data

4. Data/Computation Management
Persistent Geometric/Mesh Data
Geometry: Vertices, Edges, Faces, regions (Shephard,Flaherty …)
Mesh: Nodes, Elements, Edges(?), Faces(?)
Dynamic Computational Data: Matrices, Intermediate Solution Vectors
Matrices generated as additive blocks following the distribution of the mesh
Vectors follow distribution of nodes

5. Data/Computation Management
Data Distribution/Scheduling is achieved by:
Assigning a key to each object that is derived from the data itself
These keys define a simple ordering scheme for the data
Partitioning the key space produces a data/computation distribution

6. Space Filling Curve
hn is continuous
hn can completely fill a unit n dimensional hypercube
there exists
For any set of points

7. If points in the n-dimensional hyper-cube are close to each other then
Space Filling Curve
Characteristics of the SFC ordering (important in the case of adaptive meshes):
Geometric locality:
If points in the n-dimensional hyper-cube are close to each other then
the images under are also close to each other in the mean sense
-- can help cache performance too
Sub-cube property:
If the entire domain split up in a recursive fashion, the curve passes through
all points in each sub-cube at a particular level before going through
points in a neighboring sub-cube.
Self similarity Fractals
The curve can be generated from a basic stencil
Possible future use for integrating GIS and simulation data

8. Space Filling Curve
A unique key (identifier) of an object can be created from the SFC
The key is stored as an array of size keylength of unsigned integers
Map the location from [0,1]d to the key space K[0,Maximum Unsigned Integer*keylength]
For the key, the left most bit is the most important and the right most bit is the least important
Objects with keys that have “close” values for the leading digits of a key are near each other and objects that do not have “close” values for the leading digits of a key will likely be far from each other

9. Mesh Partitioning
To achieve good parallel efficiency computational load should
be equally distributed (if computational load changes dynamically, mesh partitioning needs to be done dynamically as well)
Good partitioning: load balanced; communication minimized
(all the processors are equally used); (communication takes a lot of time)
Two basic types of partitioning are based on:
graph (connectivity) of the mesh
Mesh quality
Cost
High
High
Geometric/mesh traversal
Mesh quality
Cost
Fair
Low

10. Mesh Partitioning and Repartitioning
The Space Filling Curve (SFC) based algorithm
Geometric mesh partitioning algorithm
IDEA: Use the SFC ordering because it is easier to deal with ordering objects and splitting sorted list in 1 dimension then in dimensions greater than 1
Determine the key of each element given by the SFC algorithm
Sort the list of objects by their keys
Divide the list to P equal pieces

11. Mesh Partitioning and Repartitioning
Problem
Repartitioning causes massive data migration and loss of parallel efficiency
Solution
Predictive load balancing strategies are used to compute incremental modifications! Load balancing is performed before mesh is adapted based on expected amount of work after mesh adaptation.

12. Predictive Load Balancing

13. Hashtable
Hashtable is used to access objects quickly while decreasing memory usage
Hashtable size should be much larger than the amount of objects that are accessed through the hashtable

14. Hashing
Object is put in hashtable according to its SFC key
Address calculator finds an object’s address from the SFC key and the minimum and maximum key for a given processor
Minimum and maximum key values are calculated at creation of the hashtable – possible to have an object that has a key less than the minimum key or greater than the maximum key, then object will be put in the beginning or end of the hashtable, respectively
Objects with larger key values are put in after objects with smaller key values
If the particular place in the hashtable is already occupied, objects at that level are stored in linked list

15. AFEAPI Hashtable

16. Code Structure

17. 1. Create mesh file (preferable using Hypermesh)
Using AFEAPI
Instructions at:
(use version AFEAPI_VBR_04.tar.gz)
1. Preprocessing
1. Create mesh file (preferable using Hypermesh)
2. Run serial preprocessing code specifying material properties and amount of processors to use
2. Run parallel code
3. Serial postprocessing in Tecplot

18. Code Structure
Set up MPI (Initialize, create MPI communication structures)
Read in data/Create persistent data (eg. Hashtable)
Create local and global ordering of dof
Solve
Calculate sparse storage info (VBR sparse storage scheme)
Assemble stiffness matrices (eliminate bubble dof if they exist)
Solve global system (reconstruct bubble dof if they were eliminated)
Postprocess
Calculate constrained nodes
Put solution in node objects
Calculate error estimate
Create results file for use in Tecplot
If error estimate is below desired tolerance: end, otherwise, go on to 7
Perform predictive load balancing/mesh partitioning
Refine the element size (h adapt)
Refine the element polynomial order (p adapt)
Smooth load balance/mesh partitions
Go to Step 3

19. Can be recycled from old FORTRAN codes!!
Code Customization
Customization for adaptive static hp-FEM requires providing routines to compute element stiffness and error e.g.
subroutine elemcom(ifg, nequ, ndff, Nc, Norder, Nelb, bcvalue, Icon, xnod, ek, ef)
subroutine errest(nequ, Norder, xnod, Utemp, Nelb, bcvalue, Icon, errorsq, solsq)
Can be recycled from old FORTRAN codes!!
For dynamic calculations or other discretization methods, customization may be a little mover involved

20. Important C++ Classes in AFEAPI
3 major classes: node class, element class, hashtable class
HASHTABLE class (csrc/header/hashtab.h):
Used for accessing node and element objects
NODE class (csrc/header/node.h):
Key is generated from node coordinates
3 Types of nodes: vertex, edge and bubble
ELEMENT Class (csrc/header/element2.h)
Only use/allow quadrilateral elements
Geometry is defined by 9 nodes
Element key is generated from bubble node coordinates
Node ordering is counterclockwise