Casper An Asynchronous Progress Model for MPI RMA on Many-core Architectures Min Si[1][2], Antonio J. Peña[1], Jeff Hammond[3], Pavan Balaji[1], Masamichi Takagi[4], Yutaka Ishikawa[4] [1] Argonne National Laboratory, USA {msi, apenya, balaji}@mcs.anl.gov [3] Intel Labs, USA jeff_hammond@acm.org Have worked at Argonne for 1 year, [2] University of Tokyo, Japan msi@il.is.s.u-tokyo.ac.jp [4] RIKEN AICS, Japan {masamichi.takagi ,yutaka.ishikawa}@riken.jp

Irregular Computations Organized around dense vectors or matrices Regular data movement pattern, use MPI SEND/RECV or collectives More local computation, less data movement Example: stencil computation, matrix multiplication, FFT* Irregular computations Organized around graphs, sparse vectors, more “data driven” in nature Data movement pattern is irregular and data-dependent Growth rate of data movement is much faster than computation Example: social network analysis, bioinformatics Increasing trend of applications are moving to irregular computation models Need more dynamic communication model * FFT : Fast Fourier Transform * The primary contents of this slide are contributed by Xin Zhao.

Message Passing Models Two-sided communication One-sided communication (Remote Memory Access) Process 0 Process 1 Send (data) Receive (data) Process 0 Process 1 += Put (data) Get (data) Acc (data) Computation Feature: Origin (P0) specifies all communication parameters Target (P1) does not explicitly receive or process message Is communication always asynchronous ?

Problems in Asynchronous Progress One-sided operations are not truly one-sided In most platforms (e.g., InfiniBand, Cray) Some operations are hardware supported (e.g., contiguous PUT/GET) Other operations have to be done in software (e.g., 3D accumulates of double precision data) Process 0 Process 1 Software implementation of one-sided operations means that the target process has to make an MPI call to make progress. MPI call Acc(data) += Computation Delay Not TRULY asynchronous ! * RDMA : Remote Direct Memory Access

Traditional Approach of ASYNC Progress (1) Thread-based approach Every MPI process has a communication dedicated background thread Background thread polls MPI progress in order to handle incoming messages for this process Example: MPICH default asynchronous thread, SWAP-bioinformatics Cons: Waste half of computing cores or oversubscribe cores Overhead of Multithreading safety of MPI Process 0 Process 1 Helper thread Computation Acc(data) += P0 T0 P1 T1 P2 T2 P3 T3

Traditional Approach of ASYNC Progress (2) Interrupt-based approach Assume all hardware resources are busy with user computation on target processes Utilize hardware interrupts to awaken a kernel thread and process the incoming RMA messages i.e., Cray MPI, IBM MPI on Blue Gene/P Cons: Overhead of frequent interrupts Process 0 Process 1 Computation Helper thread Acc(data) += Interrupt DMMAP-based ASYNC overhead on Cray XC30

Outline Background & Problem statement Existing Approaches Our solution : CASPER Ensuring Correctness and Performance Evaluation

Casper Process-based ASYNC Progress Multi- and many-core architectures Rapidly growing number of cores Not all of the cores are always keeping busy Process-based asynchronous progress Dedicating arbitrary number of cores to “ghost processes” Ghost process intercepts all RMA operations to the user processes P0 P1 P2 PN … G0 G1 Pros: No overhead caused by multithreading safety or frequent interrupts Flexible core deployment Portable PMPI* redirection Process 0 Process 1 += Computation Acc(data) Ghost Process Infinite cores, do not need interrupt Process 0 Process 1 += Computation Acc(data) MPI call Original communication Communication with Casper * PMPI : name-shifted profiling interface of MPI

Internal Memory mapping Basic Design of Casper Three primary functionalities Transparently replace MPI_COMM_WORLD by COMM_USER_WORLD Shared memory mapping between local user and ghost processes by using MPI-3 MPI_Win_allocate_shared* G 0 1 2 3 4 COMM_USER_WORLD 0 1 2 MPI_COMM_WORLD Redirect RMA operations to ghost processes Ghost Process P1 P2 P1 offset P2 offset Internal Memory mapping P0 P1 += ACC(P1, disp, user_win) Ghost Process for P1 ACC(G0, P1_offset + disp, internal_win) Computation Lock(P1) Lock(G0) lock Recv * MPI_WIN_ALLOCATE_SHARED : Allocates window that is shared among all processes in the window’s group, usually specified with MPI_COMM_TYPE_SHARED communicator.

Ensuring Correctness and Performance Asynchronous progress Correctness Performance Casper MPICH Applications Correctness challenges Lock Permission Management Self Lock Consistency Managing Multiple Ghost Processes Multiple Simultaneous Epochs CrayMPI MVAPICH …

RMA synchronization modes Active-target mode Both origin and target issue synchronization Fence (like a global barrier) PSCW (subgroup of Fence) Passive-target mode Only origin issues synchronization Lock_all (shared) Lock (shared or exclusive) P1 P0 Fence(win) P2 PUT P1 P0 P2 Lock_all(win) Unlock_all(win) PUT P1 P0 Start(P1, win) Post(P0 & P2, win) Wait(P0 & P2, win) Comp(P1, win) PUT P2 P1 P0 Unlock(P1) Lock(P1) P2 PUT

Two origins access two targets sharing the same ghost process [Correctness Challenge 1] Lock Permission Management for Shared Ghost Processes (1) Two origins access two targets sharing the same ghost process An origin accesses two targets sharing the same ghost process [POOR PERF.] Two concurrent lock epochs have to be serialized P0 P1 G0 P2 P3 G1 P2 P3 P2 Lock (P0, win) Unlock(P0, win) Lock (P1, win) Unlock(P1, win) Lock (G0, win) Unlock(G0, win) P3 Serialized Lock (G0, win) Unlock(G0, win) [INCORRECT] Nested locks to the same target P0 P1 G0 P2 P3 G1 Lock (P0, win) Lock (P1, win) Unlock(P0, win) Unlock(P1, win) Lock (G0, win) … MPI standard: An origin cannot nest locks to the same target

User hint optimization [Correctness Challenge 1] Lock Permission Management for Shared Ghost Processes (2) Solution N Windows N = max number of processes on every node COMM. to ith user process on each node goes to ith window User hint optimization Window info “epochs_used” (fence|pscw|lock|lockall by default) If “epochs_used” contains “lock”, create N windows Otherwise, only create a single window WIN[0] P0 P1 G0 P2 P3 G1 1 2 WIN[1] P0 P1 G0 P2 P3 G1 1 2 P0 P1 G0 P2 P3 G1 Such overlapping windows allow Casper to carefully bypass the lock permission management in MPI when accessing different target processes, but still take advantage of them while accessing the same target. While this approach ensures correctness, it can be expensive both for resource usage and performance.

[Correctness Challenge 2] Self Lock Consistency (1) P0 P0 P1 G0 MPI standard: Local lock must be acquired immediately Lock (P0, win) x=1 y=2 … Unlock(P0, win) Lock (G0, win) Unlock(G0, win) MPI standard: Remote lock may be delayed..

[Correctness Challenge 2] Self Lock Consistency (2) Solution (2 steps) Force-lock with HIDDEN BYTES* Lock self User hint optimization Window info no_local_loadstore Do not need both 2 steps Epoch assert MPI_MODE_NOCHECK Only need the 2nd step Lock (G0, win) Get (G0, win) Flush (G0, win) // Lock is acquired // memory barrier for managing // memory consistency Lock (P0, win) * MPI standard defines unnecessary restriction on concurrent GET and accumulate. See MPI Standard Version 3.0 , page page 456, line 39.

[Correctness Challenge 3] Managing Multiple Ghost Processes (1) Lock permission among multiple ghost processes [INCORRECT] Two EXCLUSIVE locks to the same target may be concurrently acquired Lock (EXCLUSIVE, P0, win) PUT(P0) Unlock(P0, win) P2 P0 P1 G0 P2 P3 G2 G1 G3 Lock (EXCLUSIVE, P0, win) PUT(P0) Unlock(P0, win) P3 Serialized Lock (EXCLUSIVE, G0, win) Lock (EXCLUSIVE, G1, win) // get G0 G = randomly_pick_ghost(); PUT(G) … P2 Lock (EXCLUSIVE, G0, win) Lock (EXCLUSIVE, G1, win) // get G1 G = randomly_pick_ghost(); PUT(G) … P3 MPI implementation is allowed to ignore the lock if target does not receive any operation. Empty lock can be ignored, P2 and P3 may concurrently acquire lock on G0 and G1

[Correctness Challenge 3] Managing Multiple Ghost Processes (2) Ordering and Atomicity constraints for Accumulate operations [INCORRECT] Ordering and Atomicity cannot be maintained by MPI among multiple ghost processes P0 P1 G0 G1 x = 2 GET_ACC (x, y, P1) ACC (x, P1) MPI standard: Same origin && same target location accumulates must be ordered x += 2 x += 1 y=4 (correct result is 2) ACC (x, P1) MPI standard: Concurrent accumulates to the same target location must be atomic per basic datatype element. read x = 5 x=5+1 ACC (x, P1) read x = 5 x=5+2

[Correctness Challenge 3] Managing Multiple Ghost Processes (3) Solution (2 phases) Static-Binding Phase Rank binding model Each user process binds to a single ghost process Segment binding model Segment total exposed memory on each node into NG chunks Each chunk binds to a single ghost process Only redirect RMA operations to the bound ghost process Fixed lock and ACC ordering & atomicity issues But only suitable for balanced communication patterns Static-Binding-Free Phase After operation + flush issued, “main lock” is acquired Dynamically select target ghost process Accumulate operations can not be “binding free” P0 P1 P2 G0 G1 P3 Static-rank-binding P0 P1 G0 G1 P3 Static-segment-binding P1 G0 G1 Optimization for dynamic communication patterns Lock(G0, G1) PUT(G0) Flush(G0) Acquired main lock Binding Free Unlock(G0, G1)

[Correctness Challenge 4] Multiple Simultaneous Epochs – Active Epochs (1) Simultaneous fence epochs on disjoint sets of processes sharing the same ghost processes [INCORRECT] Deadlock ! P0 P1 P2 P3 Fence(win0) Epoch 1 Fence(win1) Fence(win0) Epoch 2 Fence(win1) P0 P1 G0 P2 P3 G1 Blocked Fence(win0) Fence(win1) Blocked DEADLOCK !

[Correctness Challenge 4] Multiple Simultaneous Epochs - Active Epochs (2) Solution Every user window has an internal “global window” Translate to passive-target mode Fence PSCW Flush + Send-Receive Performed on user processes Win_allocate Fence(win0) PUT(P) … Fence(win) Win_free Win_allocate Lock_all (global win) Flush_all (global win) + Barrier(COMM_USER_WORLD) + Win_sync PUT(G) … Unlock_all (global win) Win_free [Performance issue 1] [Performance issue 3] User hint MPI_MODE_NOPRECEDE avoids it [Performance issue 2] Only say translation, and have perf issues but solved by user hints User hint (NOSTORE & NOPUT & NOPRECEDE) avoids it

Evaluation Asynchronous Progress Microbenchmark NWChem Quantum Chemistry Application Experimental Environment NERSC's newest supercomputer * Cray XC30 * https://www.nersc.gov/users/computational-systems/edison/configuration/

Asynchronous Progress Microbenchmark RMA implementation in Cray MPI v6.3.1 Test scenario Lock_all (win); for (dst=0; dst<nproc; dst++) { OP(dst, double, cnt = 1, win); Flush(dst, win); busy wait 100us; /*computing*/ } Unlock_all (win) HW-handled OP ASYNC. mode Original mode NONE Thread DMAPP mode Contig. PUT/GET Interrupt Accumulate on Cray XC30 (SW) DMAPP (Interrupt-based async) PUT on Cray XC30 (HW in DMAPP mode) DMAPP (HW PUT) Focus on cray Casper provides asynchronous progress for SW-handled ACC. Casper does not affect the performance of HW PUT

NWChem Quantum Chemistry Application (1) Computational chemistry application suite composed of many types of simulation capabilities. ARMCI-MPI (Portable implementation of Global Arrays over MPI RMA) Focus on most common used CC (coupled-cluster) simulations in a C20 molecules for i in I blocks: for j in J blocks: for k in K blocks: GET block a from A GET block b from B c += a * b /*computing*/ end do ACC block c to C Accumulate block c GET block b GET block a Real user application -> production application Perform DGEMM in local buffer Get-Compute-Update model

Evaluation 2. NWChem Quantum Chemistry Application (2) CCSD iteration in CCSD task Input data file : tce_c20_triplet Platform Configuration: 12-core Intel "Ivy Bridge" (24 cores per node) Carbon C20 Casper ASYNC. Progress helps CCSD performance Core deployment (T) portion in CCSD (T) task # COMP. # ASYNC. Original MPI 24 Casper 20 4 Thread-ASYNC (oversubscribed) Thread-ASYNC (dedicated ) 12 More compute-intensive than CCSD, more improvement

Summary Casper: a process-based asynchronous progress model MPI RMA communication is not truly one-sided Still need asynchronous progress Additional overhead in thread / interrupt-based approaches Multi- / Many-Core architectures Number of cores is growing rapidly, some cores are not always busy Casper: a process-based asynchronous progress model Dedicating arbitrary number of cores to ghost processes Mapping window regions from user processes to ghost processes Redirecting all RMA SYNC. & operations to ghost processes Linking to various MPI implementation through PMPI transparent redirection Download slides: http://sudalab.is.s.u-tokyo.ac.jp/~msi/pdf/ipdps2015-casper-slides.pdf

Backup slides

[Correctness Challenge 4] Multiple Simultaneous Epochs – Lock_all (1) Lock_all only Same translation as that for Fence lock_all in win_allocate, flush_all in unlock_all [INCORRECT] Lock_all and EXCLUSIVE lock on the same window may be concurrently acquired Lock_all (win) PUT(P0) Unlock_all(win) P2 P0 P1 G0 P2 P3 G1 Lock (EXCLUSIVE, P0, win) PUT(P0) Unlock(P0, win) P3 Serialized P2 Lock (EXCLUSIVE, G0, win[0]) PUT(G0) … P3 Lock_all (global win) PUT(G0) Unlock_all(global win) Locks may be acquired concurrently

[Correctness Challenge 3] Multiple Simultaneous Epochs – Lock_all (2) Solution Translate lock_all to a series of locks to all ghost processes P0 P1 G0 P2 P3 G1 Lock_all (win) … P2 Lock (SHARED, G0, win[0]) Lock (SHARED, G0, win[1]) Lock (SHARED, G1, win[0]) Lock (SHARED, G1, win[1]) … P2 // lock P0 // lock P1 // lock P2 // lock P3

[Performance Challenge] Memory Locality Casper internally detects the location of the user processes Only bind the closest ghost processes i.e., P0-2 are bound to G0, P3-5 are bound to G1 Node Node Socket 0 Socket 1 Socket 0 Socket 1 P0 P1 P4 G0 P0 G0 P3 G1 P2 P3 P5 G1 P1 P2 P4 P5 Memory Memory Memory Memory

Performance Issues and optimizations in Fence Win_allocate Fence(win0) PUT(P) … Fence(win) Win_allocate Lockall(global win) Flushall(global win) + MPI_barrier + Win_sync PUT(G) … [Performance issue 1] Fence: local completion of operations issued by this process (as an origin) Flushall + MPI_barrier : remote completion of all operations issued by all processes [Performance issue 3] Memory consistency management User hint MPI_MODE_NOPRECEDE avoid flushall [Performance issue 2] Always need sync between processes for Remote update before fence with local read after fence Local write before fence with remote get after fence Fence PUT y Read y Write x Get x P0 P1 User hint (NOSTORE & NOPUT & NOPRECEDE) avoid barrier

Evaluation. Asynchronous Progress Microbenchmark One-sided communication on InfiniBand cluster (Contiguous PUT/GET are handled in hardware) Accumulate (SW) PUT (HW)