An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes FLASH Tutorial May 13, 2004 Parallel Computing and MPI

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago What is Parallel Computing ? And why is it useful qParallel Computing is more than one cpu working together on one problem qIt is useful when q Large problem, could take very long q Data size too big to fit in the memory of one processor qWhen to parallelize q Problem could be subdivided into relatively independent tasks qHow much to parallelize q While the speedup in computation relative to single processor is of the order of number of processors

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Parallel paradigms qSIMD – Single instruction multiple data q Processors work in lock-step qMIMD – Multiple instruction multiple data q Processors do their own thing with occasional synchronization qShared Memory q One way communications qDistributed Memory q Message passing qLoosely Coupled q When the process on each cpu is fairly self contained and relatively independent of processes on other cpu’s qTightly Coupled q When cpu’s need to communicate with each other frequently

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago How to Parallelize qDivide a problem into a set of mostly independent tasks q Partitioning a problem qTasks get their own data q Localize a task qThey operate on their own data for the most part q Try to make it self contained qOccasionally q Data may be needed from other tasks q Inter-process communication q Synchronization may be required between tasks q Global operation qMap tasks to different processors q One processor may get more than one task q Task distribution should be well balanced

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago New Code Components qInitialization qQuery parallel state q Identify process q Identify number of processes qExchange data between processes q Local, Global qSynchronization q Barriers, Blocking Communication, Locks qFinalization

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago MPI qMessage Passing Interface, standard for distributed memory model of parallelism qMPI-2 will support one-way communication, commonly associated with shared memory operations qWorks with communicators; a collection of processors q MPI_COMM_WORLD default qHas support for lowest level communication operations and composite operations qHas blocking and non-blocking operations

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Communicators COMM1 COMM2

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Low level Operations in MPI qMPI_Init qMPI_Comm_size q Find number of processors qMPI_Comm_rank q Find my processor number qMPI_Send/Recv q Communicate with other processors one at a time qMPI_Bcast q Global data transmission qMPI_Barrier q Synchronization qMPI_Finalize

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Advanced Constructs in MPI qComposite Operations q Gather/Scatter q Allreduce q Alltoall qCartesian grid operations q Shift qCommunicators q Creating subgroups of processors to operate on qUser-defined Datatypes qI/O q Parallel file operations

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Communication Patterns Collective 0123 Shift 10 2 All to All Point to Point One to All Broadcast

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Communication Overheads qLatency vs. Bandwidth qBlocking vs. Non-Blocking q Overlap q Buffering and copy qScale of communication q Nearest neighbor q Short range q Long range qVolume of data q Resource contention for links qEfficiency q Hardware, software, communication method

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Parallelism in FLASH qShort range communications q Nearest neighbor qLong range communications q Regridding qOther global operations q All-reduce operations on physical quantities q Specific to solvers q multi-pole method q FFT based solvers

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Domain Decomposition P0 P1 P2P3

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Border Cells / Ghost Points qWhen splitting up solnData, need data from other processors. qNeed a layer of cells from each processor qNeed to update each time step

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Border/Ghost Cells Short Range communication

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Two MPI Methods for doing it qMPI_Cart_create q Create topology qMPE_Decomp1d q Domain decomp on topology qMPI_Cart_shift q Who’s on the left/right? qMPI_SendRecv q Ghost cells left qMPI_SendRecv q Ghost cells right qMPI_Comm_rank qMPI_Comm_size qManually decompose grid over processors qCalculate left/right qMPI_Send/MPI_Recv q Carefully to avoid deadlocks

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Adaptive Grid Issues qDiscretization not uniform qSimple left-right guard cell fills inadequate qAdjacent grid points may not be mapped to the nearest neighbors in processors topology qRedistribution of work necessary

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Regridding qChange in number of cells/blocks qSome processors get more work than others qLoad imbalance qRedistribute data to even out work on all processors qLong range communications qLarge quantities of data moved

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Regridding

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Other parallel operations in FLASH qGlobal max/sum etc (Allreduce) q Physical quantities q In solvers q Performance monitoring qAlltoall q FFT based solver on UG qUser defined datatypes and file operations q Parallel I/O