Dynamic Load Balancing for VORPAL Viktor Przebinda Center for Integrated Plasma Studies

What is VORPAL? A parallel simulation package that models laser - plasma interactions using the PIC or fluid model. Built on an object oriented framework. Over 1,300 pages (68,000 lines) of C++ code.

Goal: Implement Dynamic Load Balancing for VORPAL Dynamic: Automatically adjusts decomposition appropriately at runtime. Fully compatible with all simulations. Efficient: Minimizes overhead. User friendly: Requires no special configuration.

Grids define how regions are to be updated. extended region physical region One or two cells wide

Extended regions simplify parallel synchronization. extended region physical region extended region physical region CPU 1CPU 0 Extended regions contain data in physical regions of other CPUs.

Allocated region provides simple over allocation scheme. extended region physical region Allocated Region

Working with current framework introduces restrictions. Boundaries can be adjusted by at most one cell in each direction. Decomposition description is restricted to prevent irregular domains. Direction 0 Direction 1 Direction 0 Direction 1 Impossible Normal

Two strategies exist to accommodate decomposition adjustment. Over allocate all fields to accommodate future decomposition adjustment. Resize and copy all field data as needed for each decomposition adjustment.

Over Allocation Region in use overflow Disadvantage: Decreases cache hits, resulting in lower efficiency Advantage: Minimal overhead. overflow

Over allocation introduces minimal overhead. Region in use overflow

Over Allocation in direction of maximum stride is optimal. Region in use overflow

Resizing New memory block requested, values are copied over Disadvantage: Large overhead Advantage: Does not affect efficiency.

Optimal performance is achieved through both methods New memory block requested, values are copied over Memory is over allocated to prevent future resizes.

Decomposition adjustment occurs at end of update UpdateLoad Balancing CPU time measured each iteration

DLB is achieved in eight steps. 1.All nodes send processing times to node zero. 2.Node zero decides whether to perform an adjustment. 3.Node zero constructs an adjustment description and sends to all other nodes. 4.All nodes apply the adjustment, reconfigure grid. 5.All fields resize as necessary. 6.Field iteraters are rebuilt 7.All messengers are rebuilt 8.Fields and particles are synchronized.

1. Processing times are collected to aid in adjustment decisions. Each node measures the virtual time it took to perform its last update. Virtual time excludes time spent blocked on I/O. This amount is sent to node zero. The process waits for reply from node zero. Time

2. Adjustment decision made based on idle time. Given the cost of performing a load balance, VORPAL only does so only if any node during the last time step was idle for more than 10% of the time. Time

3. Adjustment is computed to eliminate bottlenecks Using a greedy algorithm, node zero constructs adjustment information for all processors. This is sent to all nodes. Time

Boundaries are shrunk around processors with high load. CPU1 CPU2CPU3 CPU Load lowest highest Direction 0 Direction 1 CPU0

4. Decomposition object is modified on each processor Each node applies the adjustment. The local grid is adjusted to match the new size. The allocated region is modified if it cannot support the new size. Time

5. Fields resize if allocated region has changed. All fields check the allocated region to see if it has grown. If so, the field allocates additional memory to accommodate 25 more cells in the direction of growth. Time

6. Outdated information in Field iteraters is rebuilt. Pointers to specific memory locations may have changed if a resize was performed. Physical and extended regions may have changed size. Time

7. Outdated message passing objects are rebuilt. Intersecting regions may have changed so the must be reconstructed. Time

8. New boundaries must be synchronized with neighbors. Field data on physical boundaries is sent to neighboring processors and extended regions are filled from neighbors. Particles that may have crossed outside the boundary of the current node are sent to neighboring nodes. Unfortunately, since there is nothing to do while synchronization takes place, an enormous overhead is seen at this step. Time

When to use load balancing. When running a PIC simulations. When plasma concentration is expected to change. When decomposition is along the zero direction. When a large number of time steps are used.

Case study: DLB can beat best static decomposition by 23%. CPU0 CPU1 Particles loaded into right region Boundary at midpoint

Sliding window moves particles left to CPU0 CPU0 CPU1 Particles shifted into left region

Standard run shows large differences in CPU use.

Load Balancing ensures uptimes on each node are equal.

Conclusion Load balancing performs desired functions. Overhead involved in message passing is quite significant, somewhat limiting usefulness.