Motivation: dynamic apps Rocket center applications: –exhibit irregular structure, dynamic behavior, and need adaptive control strategies. Geometries are irregular Dynamic changes: – burning, pressurization, crack propagation,

Motivation: modularity Rocket center apps are multicomponent –Codes developed by different teams –Different discretization schemes may be used, etc. –Multiple alternative strategies may be implemented for individual components Need –to develop reusable enabling technologies

Need for adaptive strategies Computation structure changes over time: –Combustion Adaptive techniques in application codes: –Adaptive refinement in structures or even fluid –Other codes such as crack propagation Can affect the load balance dramatically –One can go from 90% efficiency to less than 25%

Multi-partition decomposition using objects: Idea: decompose the problem into a number of partitions, –independent of the number of processors –# Partitions > # Processors The system maps partitions to processors –The system should be able to map and re-map objects as needed

Supporting Multi-partition approach A Load balancing framework is needed to support such an approach: –Charm++ –Migration support –Automatic instrumentation and object-load database –Re-mapping strategies

Charm++ A parallel C++ library –Supports data driven objects singleton objects, object arrays, groups, –Many objects per processor, with method execution scheduled with availability of data –System supports automatic instrumentation and object migration –Works with other paradigms: MPI, openMP,..

Data driven execution in Charm++ Scheduler Message Q

Load Balancing Framework Aimed at handling... –Continuous (slow) load variation –Abrupt load variation (refinement) –Workstation clusters in multi-user mode Measurement based –Exploits temporal persistence of computation and communication structures –Very accurate (compared with estimation) –instrumentation possible via Charm++/Converse

Object balancing framework

Utility of the framework: workstation clusters Cluster of 8 machines, –One machine gets another job –Parallel job slows down on all machines Using the framework: –Detection mechanism –Migrate objects away from overloaded pe –Restored almost original throughput!

Utility of the framework: Intrinsic load imbalance To test the abilities of the framework –A simple problem: Gauss-Jacobi iterations –Refine selected sub-domains ConSpector: web based tool –Submit parallel jobs –Monitor performance and application behavior –Interact with running jobs via GUI interfaces

How to utilize this framework for CSE applications : Explicit programming Conversion from existing MPI programs Higher level frameworks

Supporting explicit programming: Libraries, Structured dagger Migratable threads

Conversion from existing MPI programs: Is it possible? Yes, with a couple of tricks Thread based approach No-threads approach

Thread based approach Each MPI process: –becomes a chunk, with multiple chunks per pe Each chunk implemented as a thread –within a Charm++ object Common MPI calls: –Implemented on top of Charm++/threads –Suspend threads instead of blocking Migration of threads: tricky but supported

Conversion of MPI programs: Collect all global variables: –in a "chunk" data structure –(except read-only, for efficiency) Replace each occurrence: –of such a global variable x with chunk%x Provide subroutines: – for packing and unpacking the chunk data structure into a buffer

No-threads approach Use an "irecv with continuation" library: –usual MPI style irecv + waitall(f) Conversion: –All the steps of the thread approach + –Split subroutines at “receive”s Faster context switching than threads, with some extra effort

Case studies: ROCFLO using the threads approach Crack propagation code: –with a non-thread approach Relatively non--invasive, and within reasonable effort Reviewed/in progress: –ROCSOLID, OPAAL codes

Further up: In Progress Automated conversion from MPI: –Compiler based approach FEM framework Ghost array framework Load balancing strategies Component libraries

Further up: plans Geometric Interfaces across modules –interpolation, data structures Support for on-line re-meshing Support for Visualization Parallel I/O components Orchestration: –Controlling simulation at a higher level

Charm++ Converse Load database + balancer MPI-on-CharmIrecv+ Automatic Conversion from MPI FEM Structured Cross module interpolation Higher level framework Migration path Framework path