1. Dynamic Load Balancing in Scientific Simulation
Angen Zheng

2. Static Load Balancing: No Data Dependency
Unchanged Load Distribution
Computations
No Communication among PUs.
Initial Balanced Load Distribution
Initial Load
PU 1
PU 2
PU 3
Distribute the load evenly across processing unit.
Is this good enough? It depends!
No data dependency!
Load distribution remain unchanged!
Minimize computation time by distributing the data evenly among processors.
Minimize communication time caused by cross-processor data dependencies.
However, the data distribution and communication patterns change dynamically. Initially balanced partition, may result in serial imbalanced partition as simulation proceed.
Thus, it requires periodically redistributing the data evenly across processors.

3. Static Load Balancing: Data Dependency
Unchanged Load Distribution
Computation
PUs need to communicate with each other to carry out the computation.
Initial Balanced Load Distribution
Initial Load
PU 1
PU 2
PU 3
Distribute the load evenly across processing unit.
Minimize inter-processing-unit communication!
By collocating the most communicating data into a single PU.
Minimize computation time by distributing the data evenly among processors.
Minimize communication time caused by cross-processor data dependencies.
However, the data distribution and communication patterns change dynamically. Initially balanced partition, may result in serial imbalanced partition as simulation proceed.
Thus, it requires periodically redistributing the data evenly across processors.

4. Load Balancing in Scientific Simulation
PUs need to communicate with each other to carry out the computation.
Initial Balanced Load Distribution
Initial Load
PU 1
PU 2
PU 3
Imbalanced Load Distribution
Iterative Computation Steps
Balanced Load Distribution
Repartitioning
Distribute the load evenly across processing unit.
Minimize inter-processing-unit communication!
By collocating the most communicating data into a single PU.
Minimize data migration among processing units.
Minimize computation time by distributing the data evenly among processors.
Minimize communication time caused by cross-processor data dependencies.
However, the data distribution and communication patterns change dynamically. Initially balanced partition, may result in serial imbalanced partition as simulation proceed.
Thus, it requires periodically redistributing the data evenly across processors.
Dynamic Load Balancing

5. Dynamic Load Balancing: (Hyper)graph Partitioning
Given a (Hyper)graph G=(V, E).
(Hyper)graph Partitioning
Partition V into k partitions P0, P1, … Pk, such that all parts
Disjoint: P0 U P1 U … Pk = V and Pi ∩ Pj = Ø where i ≠ j.
Balanced: |Pi| ≤ (|V| / k) * (1 + ᵋ)
Edge-cut is minimized: edges crossing different parts.
Bcomm= 3
Hypergraph = (V, N)
Each net is a non-empty subset of vertices.
First, it builds the initial hypergraph for the initial simulation data.
Then, it partitions the initial hypergraph into 3 balanced partitions and maps each partition to a processor.
At each rebalancing point, it updates the initial hypergraph to capture the changes in data distribution and communication pattern.
Then, it repartition the updated hypergraph into 3 balanced partitions.
The load balancing problem in scientific simulation is usually modeled as a graph or hypergraph partitioning issue Vertices represent the computation and edges or hyperedge reflect data dependencies. Each vertex can be associated with a weight reflecting the amount of the computation, and each edge or hyperedge can be assigned a weight approximating the communication cost between adjacent nodes. By applying existing heuristic graph or hypergraph partitioning algorithms, the graph can be evenly partitioned into multiple balanced partitions while minimizing the number of edge-cut. As a result, the computation is evenly distributed across processors while the communication among different processors is minimized.

6. Dynamic Load Balancing: (Hyper)graph Repartitioning
Given a partitioned (Hyper)graph G=(V, E).
(Hyper)graph Repartitioning
Repartition V into k partitions P0, P1, … Pk, such that all parts
Disjoint.
Balanced.
Minimal Edge-cut.
Minimal Migration.
Bcomm = 4
Bmig =2
Repartitioning
Initial Partitioning

7. Dynamic Load Balancing: (Hypergraph) Repartition-Based
Building the (Hyper)graph
Vertices represent data.
Vertex object size reflects the amount of the data per vertex.
Vertex weight accounts for computation per vertex.
Edges reflects data dependencies.
Edge weight represents the communication among vertices.
Build the Initial (Hyper)graph
Load Distribution
After Repartitioning
Repartitioning the Updated (Hyper)graph
Initial Partitioning
PU1
PU2
PU3
Update the Initial (Hyper)graph
Iterative Computation Steps
Hypergraph = (V, N)
Each net is a non-empty subset of vertices.
First, it builds the initial hypergraph for the initial simulation data.
Then, it partitions the initial hypergraph into 3 balanced partitions and maps each partition to a processor.
At each rebalancing point, it updates the initial hypergraph to capture the changes in data distribution and communication pattern.
Then, it repartition the updated hypergraph into 3 balanced partitions.
The load balancing problem in scientific simulation is usually modeled as a graph or hypergraph partitioning issue Vertices represent the computation and edges or hyperedge reflect data dependencies. Each vertex can be associated with a weight reflecting the amount of the computation, and each edge or hyperedge can be assigned a weight approximating the communication cost between adjacent nodes. By applying existing heuristic graph or hypergraph partitioning algorithms, the graph can be evenly partitioned into multiple balanced partitions while minimizing the number of edge-cut. As a result, the computation is evenly distributed across processors while the communication among different processors is minimized.
Reduce the Dynamic Load Balancing to a (Hyper)graph Repartitioning Problem.

8. (Hypergraph) Repartition-Based Dynamic Load Balancing: Cost Model
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.

9. (Hypergraph) Repartition-Based Dynamic Load Balancing: Network Topology
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.

10. (Hypergraph) Repartition-Based Dynamic Load Balancing: Cache-Hierarchy
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.
Initial (Hyper)graph
Initial Partitioning
PU1
PU2
PU3
Updated (Hyper)graph
Iterative Computation Steps
Migration Once After Repartitioning
Rebalancing

11. Hierarchical Topology-Aware (Hyper)graph Repartition-Based Dynamic Load Balancing
Inter-Node Repartitioning:
Goal: Group the most communicating data into compute nodes closed to each other.
Solution:
Regrouping.
Repartitioning.
Refinement.
Intra-Node Repartitioning:
Goal: Group the most communicating data into cores sharing more level or caches.
Solution#1: Hierarchical repartitioning.
Solution#2: Flat repartitioning.
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.

12. Hierarchical Topology-Aware (Hyper)graph Repartition-Based Dynamic Load Balancing
Inter-Node Repartitioning
Regrouping.
Repartitioning.
Refinement.
Regrouping
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.

13. Hierarchical Topology-Aware (Hyper)graph Repartition-Based Dynamic Load Balancing
Inter-Node (Hyper)graph Repartitioning
Regrouping.
Repartitioning.
Refinement.
Repartitioning
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.
Migration Cost: 2 (inter-node) + 2 (intra-node)
Communication Cost: 3 (inter-node)

14. Topology-Aware Inter-Node (Hyper)graph Repartitioning
Regrouping.
Repartitioning.
Refinement.
Refinement
They all assume that communication and migration time is proportional to the volume of the data communicated and migrated.
Migration Cost: 2 (intra-node)
Communication Cost: 3 (inter-node)

15. Hierarchical Topology-Aware Intra-Node (Hyper)graph Repartitioning
Main Idea: Repartition the subgraph assigned to each node hierarchically according to the cache hierarchy. 2 3 4 5 1 1 2 3 4 5

16. Flat Topology-Aware Intra-Node (Hyper)graph Repartition

17. Flat Topology-Aware Intra-Node (Hyper)graph Repartition
Core#0
Core#1
Core#2
Old Partition Assignment
Old Partition

18. Flat Topology-Aware Intra-Node (Hyper)graph Repartition
Old Partition
New Partition

19. Flat Topology-Aware Intra-Node (Hyper)graph Repartition
Core#0
Core#1
Core#2
Old Partition Assignment
Core#0
Core#1
Core#2
Core#3 P1 4 P2 2 P3 P4
Partition Migration Matrix P1 P2 P3 P4 1 3
New Partition
Partition Communication Matrix

20. Flat Topology-Aware Intra-Node (Hyper)graph Repartition
Core#1
Core#2
Core#3
Core#4 P1 4 P2 2 P3 P4
Partition Migration Matrix P1 P2 P3 P4 1 3
New Partition
Partition Communication Matrix P1 P2 P3 P4
Core#0
Core#1
Core#2
Core#3

21. Major References
[1] K. Schloegel, G. Karypis, and V. Kumar, Graph partitioning for high performance scientific simulations. Army High Performance Computing Research Center, 2000.
[2] B. Hendrickson and T. G. Kolda, Graph partitioning models for parallel computing," Parallel computing, vol. 26, no. 12, pp. 1519~1534, 2000.
[3] K. D. Devine, E. G. Boman, R. T. Heaphy, R. H.Bisseling, and U. V. Catalyurek, Parallel hypergraph partitioning for scientific computing," in Parallel and Distributed Processing Symposium, IPDPS th International, pp. 10-pp, IEEE, 2006.
[4] U. V. Catalyurek, E. G. Boman, K. D. Devine,D. Bozdag, R. T. Heaphy, and L. A. Riesen, A repartitioning hypergraph model for dynamic load balancing," Journal of Parallel and Distributed Computing, vol. 69, no. 8, pp. 711~724, 2009.
[5] E. Jeannot, E. Meneses, G. Mercier, F. Tessier,G. Zheng, et al., Communication and topology-aware load balancing in charm++ with treematch," in IEEE Cluster 2013.
[6] L. L. Pilla, C. P. Ribeiro, D. Cordeiro, A. Bhatele,P. O. Navaux, J.-F. Mehaut, L. V. Kale, et al., Improving parallel system performance with a numa-aware load balancer," INRIA-Illinois Joint Laboratory on Petascale Computing, Urbana, IL, Tech. Rep. TR-JLPC-11-02, vol , 2011.

22. Thanks!