1. Martin Berzins (Steve Parker) What are the hard apps problems? How do the solutions get shared? What non-apps work is needed? Thanks to DOE for funding since 1997, NSF since 2008 Uintah solver for multiphase-fluid-structure interaction problems- explosive filled container in fire

2. Martin Berzins (Steve Parker) What are the hard apps problems? ADAPTIVE DYNAMIC GLOBAL How do the solutions get shared? ENCAPSULATION ABSTRACTION What non-apps work is needed? APPLICATION DRIVEN TOOLS Thanks to DOE for funding since 1997, NSF since 2008 Uintah solver for multiphase-fluid-structure interaction problems- explosive filled container in fire

3. Hard Apps Problems are Multi- Physics/Multiscale with Adaptive Methods and/or Global Comms. 1.Lack of predictability forces use of dynamic load balancing methods. 2.AMR data structures – migration. 3.Radiation problems –global communications needed. 4.Particles move across grid with our methods 5.Any attempt to compute to a particular solution accuracy will need adaptive methods UPREDICTABLE!!

4. Particle Variables Cell Centered Variables Cell –Vertex Variables Fundamental Uintah data Structure is a patch – multiple variable types SFC Load balancing uses patches Uintah Domain Decomposition User writes code for a patch and its communications only - Uintah uses this information to construct communications pattern via a task graph

5. Combining MPM and Mesh refinement

6. How Does Uintah Work? Simulation Controller Simulation Controller Problem Specification Problem Specification XML Simulation (One of Arches, ICE, MPM, MPMICE, MPMArches, …) Simulation (One of Arches, ICE, MPM, MPMICE, MPMArches, …) Scheduler Tasks Data Archiver Data Archiver Tasks Callbacks MPI Assignments Load Balancer Load Balancer Configuration

7. Burgers Equation code void Burger::timeAdvance(const ProcessorGroup*, const PatchSubset* patches, const MaterialSubset* matls, DataWarehouse* old_dw, DataWarehouse* new_dw) { //Loop for all patches on this processor for(int p=0;p size();p++){ //Get data from data warehouse including 1 layer of "ghost" nodes from surrounding patches old_dw->get(u, lb_->u, matl, patch, Ghost::AroundNodes, 1); // dt, dx Time and space increments Vector dx = patch->getLevel()->dCell(); old_dw->get(dt, sharedState_->get_delt_label()); // allocate memory for results new_dw->allocateAndPut(new_u, lb_->u, matl, patch); // define iterator range l and h …… lots missing here and Iterate through all the nodes for(NodeIterator iter(l, h);!iter.done(); iter++){ IntVector n = *iter; double dudx = (u[n+IntVector(1,0,0)] - u[n-IntVector(1,0,0)]) /(2.0 * dx.x()); double du = - u[n] * dt * (dudx); new_u[n]= u[n] + du; }

8. 8 Task graph Each algorithm defines a description of the computation –Required inputs and outputs (names and spatial relationships) –Callbacks to perform each task on a single subregion of space Communication is performed at the edges in the graph Uintah uses this information to create a graph of computation and communication

9. AMR Scalability Challenging dynamically changing workload 8x8x8 patches: timings over about 30 timesteps 8K to 20K patches Original remeshes at every step Dilated only every 4 steps or so

10. Atlas LLNL 1152 nodes of 4 Opteron dual-core Infiniband Thunder LLNL 1024 nodes of 4 Intel Itainum2 Quadrics switch Redstorm SNL 13284 nodes of Opteron dual-core XT3 Ranger UT Austin 3,936 nodes of 4 Barcelona quad-core Infiniband Small Problem only 2-3 patches per proc at 4096 procs

11. Summary Uintah is adaptive multi-physics AMR code Clear separation between application user and system components Very general CS approach to load balancing and scalability Expensive multidisciplinary effort.

13. Existing AMR scalability work Brian Van Stralen: weak scaling easy strong scaling hard Strong scaling problems include message overload, malloc inconsistencies and load imbalance. Not clear that these problems cannot be solved AMR does scale to 12K processors Flash Code BG/L and beyond (?) Strong scalability: fixed problem size doubling processors should halve execution time Weak scalability: problem size grows with processors Doubling processors give constant execution time