1. AMPI: Adaptive MPI Tutorial
Gengbin Zheng
Parallel Programming Laboratory
University of Illinois of Urbana-Champaign

2. Motivation Challenges AMPI: Adaptive MPI
2018/9/17
Motivation
Challenges
New generation parallel applications are:
Dynamically varying: load shifting, adaptive refinement
Typical MPI implementations are:
Not naturally suitable for dynamic applications
Set of available processors:
May not match the natural expression of the algorithm
AMPI: Adaptive MPI
MPI with virtualization: VP (“Virtual Processors”)
Hotspots of computing move over the time. Causing load imbalance.
Wrapping around if not enough PEs
9/17/2018
CS420

3. Outline Future work MPI basics Charm++/AMPI introduction
How to write AMPI programs
Running with virtualization
How to convert an MPI program
Using AMPI extensions
Automatic load balancing
Non-blocking collectives
Checkpoint/restart mechanism
Interoperability with Charm++
ELF and global variables
Future work
9/17/2018
CS420

4. MPI Basics Standardized message passing interface
2018/9/17
MPI Basics
Standardized message passing interface
Passing messages between processes
Standard contains the technical features proposed for the interface
Minimally, 6 basic routines:
int MPI_Init(int *argc, char ***argv) int MPI_Finalize(void)
int MPI_Comm_size(MPI_Comm comm, int *size) int MPI_Comm_rank(MPI_Comm comm, int *rank)
int MPI_Send(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) int MPI_Recv(void* buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)
Forum founded in 1993, released Standard 1.0 in 1994, 1.1 in 1995, 2 in 1997 by vote.
Our dept head Marc Snir is a major contributor to the standard!
Status containing error info for (potentially) multiple requests.
9/17/2018
CS420

5. MPI Basics MPI-1.1 contains 128 functions in 6 categories:
2018/9/17
MPI Basics
MPI-1.1 contains 128 functions in 6 categories:
Point-to-Point Communication
Collective Communication
Groups, Contexts, and Communicators
Process Topologies
MPI Environmental Management
Profiling Interface
Language bindings: for Fortran, C
20+ implementations reported
C++ binding is not mentioned till 2.0.
9/17/2018
CS420

6. MPI Basics MPI-2 Standard contains:
2018/9/17
MPI Basics
MPI-2 Standard contains:
Further corrections and clarifications for the MPI-1 document
Completely new types of functionality
Dynamic processes
One-sided communication
Parallel I/O
Added bindings for Fortran 90 and C++
Lots of new functions: 188 for C binding
C++ binding is not mentioned till 2.0.
9/17/2018
CS420

7. AMPI Status Compliance to MPI-1.1 Standard Partial MPI-2 support
Missing: error handling, profiling interface
Partial MPI-2 support
One-sided communication
ROMIO integrated for parallel I/O
Missing: dynamic process management, language bindings
9/17/2018
CS420

8. MPI Code Example: Hello World!
#include <stdio.h>
#include <mpi.h>
int main( int argc, char *argv[] ) {
int size,myrank;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
printf( "[%d] Hello, parallel world!\n", myrank );
MPI_Finalize();
return 0; }
[Demo: hello, in MPI…]
9/17/2018
CS420

9. Another Example: Send/Recv
...
double a[2], b[2];
MPI_Status sts;
if(myrank == 0){
a[0] = 0.3; a[1] = 0.5;
MPI_Send(a,2,MPI_DOUBLE,1,17,MPI_COMM_WORLD);
}else if(myrank == 1){
MPI_Recv(b,2,MPI_DOUBLE,0,17,MPI_COMM_WORLD,&sts);
printf(“[%d] b=%f,%f\n”,myrank,b[0],b[1]); }
[Demo: later…]
9/17/2018
CS420

10. Outline Future work MPI basics Charm++/AMPI introduction
How to write AMPI programs
Running with virtualization
How to convert an MPI program
Using AMPI extensions
Automatic load balancing
Non-blocking collectives
Checkpoint/restart mechanism
Interoperability with Charm++
ELF and global variables
Future work
9/17/2018
CS420

11. Charm++ Basic idea of processor virtualization
2018/9/17
Charm++
Basic idea of processor virtualization
User specifies interaction between objects (VPs)
RTS maps VPs onto physical processors
Typically, # virtual processors > # processors
System implementation
User View
Number of chares:
Independent of number of processors
Typically larger than number of processors
Seeking optimal division of labor between “system” and programmer:
Decomposition done by programmer, everything else like mapping and scheduling is automated
9/17/2018
CS420

12. Charm++ Charm++ characteristics Charm++ features explored by AMPI
Data driven objects
Asynchronous method invocation
Mapping multiple objects per processor
Load balancing, static and run time
Portability
Charm++ features explored by AMPI
User level threads, do not block CPU
Light-weight: context-switch time ~ 1μs
Migratable threads
9/17/2018
CS420

13. AMPI: MPI with Virtualization
2018/9/17
AMPI: MPI with Virtualization
Each virtual process implemented as a user-level thread embedded in a Charm++ object
Real Processors
MPI “processes”
Implemented as virtual processes (user-level migratable threads)
MPI processes
Implemented as virtual processors (user-level migratable threads)
9/17/2018
CS420

14. Comparison with Native MPI
2018/9/17
Comparison with Native MPI
Performance
Slightly worse w/o optimization
Being improved, via Charm++
Flexibility
Big runs on any number of processors
Fits the nature of algorithms
27, 64, 125, 216, 512
Two points in comparison:
Performance is slightly worse:
1) we are running on top of MPI
2) remember more advantages AMPI are not used here.
2. Number of processors flexibility
Some overhead of offset by the caching effect, etc.
Problem setup: 3D stencil calculation of size 2403 run on Lemieux.
AMPI runs on any # of PE’s (eg 19, 33, 105). Native MPI needs P=K3
9/17/2018
CS420

15. Building Charm++ / AMPI
2018/9/17
Building Charm++ / AMPI
Download website:
Please register for better support
Build Charm++/AMPI
> ./build <target> <version> <options> [charmc-options]
To build AMPI:
> ./build AMPI net-linux -g (-O3)
Version includes your communication network and OS
Charmc-options like -g
9/17/2018
CS420

16. Outline Future work MPI basics Charm++/AMPI introduction
How to write AMPI programs
Running with virtualization
How to convert an MPI program
Using AMPI extensions
Automatic load balancing
Non-blocking collectives
Checkpoint/restart mechanism
Interoperability with Charm++
ELF and global variables
Future work
9/17/2018
CS420

17. How to write AMPI programs (1)
2018/9/17
How to write AMPI programs (1)
Write your normal MPI program, and then…
Link and run with Charm++
Build your charm with target AMPI
Compile and link with charmc
include charm/bin/ in your path
> charmc -o hello hello.c -language ampi
Run with charmrun
> charmrun hello
Example of two threads sharing one counter.
A sets it to 2, B increments it, A reads the incorrect count.
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 20PE -> 50MB per PE
9/17/2018
CS420

18. How to write AMPI programs (2)
Now we can run most MPI programs with Charm++
mpirun –npK  charmrun prog +pK
MPI’s machinefile: Charm’s nodelist file
Demo - Hello World! (via charmrun)
9/17/2018
CS420

19. How to write AMPI programs (3)
2018/9/17
How to write AMPI programs (3)
Avoid using global variables
Global variables are dangerous in multithreaded programs
Global variables are shared by all the threads on a processor and can be changed by any of the threads
Thread 1
Thread2
count=1
block in MPI_Recv
b=count
count=2
Example of two threads sharing one counter.
A sets it to 2, B increments it, A reads the incorrect count.
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 20PE -> 50MB per PE
incorrect value is read!
9/17/2018
CS420

20. How to run AMPI programs (1)
2018/9/17
How to run AMPI programs (1)
Now we can run multithreaded on one processor
Running with many virtual processors:
+p command line option: # of physical processors
+vp command line option: # of virtual processors
> charmrun hello +p3 +vp8
Demo - Hello Parallel World!
Demo - 2D Jacobi Relaxation
Example of two threads sharing one counter.
A sets it to 2, B increments it, A reads the incorrect count.
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 20PE -> 50MB per PE
9/17/2018
CS420

21. How to run AMPI programs (2)
2018/9/17
How to run AMPI programs (2)
Multiple processor mappings are possible
> charmrun hello +p3 +vp6 +mapping <map>
Available mappings at program initialization:
RR_MAP: Round-Robin (cyclic)
BLOCK_MAP: Block (default)
PROP_MAP: Proportional to processors’ speeds
Demo – Mapping…
Example of two threads sharing one counter.
A sets it to 2, B increments it, A reads the incorrect count.
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 20PE -> 50MB per PE
9/17/2018
CS420

22. How to run AMPI programs (3)
Specify stack size for each thread
Set smaller/larger stack sizes
Notice that thread’s stack space is unique acros processors
Specify stack size for each thread with +tcharm_stacksize command line option:
charmrun hello +p2 +vp8 +tcharm_stacksize
Default stack size is 1 MByte for each thread
Demo – bigstack
Small array, many VP’s x Large array, any VP’s
9/17/2018
CS420

23. Outline Future work MPI basics Charm++/AMPI introduction
How to write AMPI programs
Running with virtualization
How to convert an MPI program
Using AMPI extensions
Automatic load balancing
Non-blocking collectives
Checkpoint/restart mechanism
Interoperability with Charm++
ELF and global variables
Future work
9/17/2018
CS420

24. How to convert an MPI program
2018/9/17
How to convert an MPI program
Remove global variables if possible
If not possible, privatize global variables
Pack them into struct/TYPE or class
Allocate struct/type in heap or stack
Original Code
MODULE shareddata
TYPE chunk
INTEGER :: myrank
DOUBLE PRECISION :: xyz(100)
END TYPE
END MODULE
AMPI Code
Avoid large stack data allocation because stack size in AMPI is fixed and doesn’t grow at run time.
Can be specified with command line option.
MODULE shareddata
INTEGER :: myrank
DOUBLE PRECISION :: xyz(100)
END MODULE
9/17/2018
CS420

25. How to convert an MPI program
2018/9/17
How to convert an MPI program
Original Code
SUBROUTINE MPI_Main
USE shareddata
USE AMPI
INTEGER :: i, ierr
TYPE(chunk), pointer :: c
CALL MPI_Init(ierr)
ALLOCATE(c)
CALL MPI_Comm_rank(
MPI_COMM_WORLD,
c%myrank, ierr)
DO i = 1, 100
c%xyz(i) = i + c%myrank
END DO
CALL subA(c)
CALL MPI_Finalize(ierr)
END SUBROUTINE
AMPI Code
PROGRAM MAIN
USE shareddata
include 'mpif.h'
INTEGER :: i, ierr
CALL MPI_Init(ierr)
CALL MPI_Comm_rank(
MPI_COMM_WORLD,
myrank, ierr)
DO i = 1, 100
xyz(i) = i + myrank
END DO
CALL subA
CALL MPI_Finalize(ierr)
END PROGRAM
Can use stack space by removing pointer and ALLOCATE, but limited stacksize
9/17/2018
CS420

26. How to convert an MPI program
Original Code
SUBROUTINE subA(c)
USE shareddata
TYPE(chunk) :: c
INTEGER :: i
DO i = 1, 100
c%xyz(i) = c%xyz(i) + 1.0
END DO
END SUBROUTINE
AMPI Code
SUBROUTINE subA
USE shareddata
INTEGER :: i
DO i = 1, 100
xyz(i) = xyz(i) + 1.0
END DO
END SUBROUTINE
C examples can be found in the AMPI manual
9/17/2018
CS420

27. How to convert an MPI program
2018/9/17
How to convert an MPI program
Fortran program entry point: MPI_Main
program pgm  subroutine MPI_Main
end program end subroutine
C program entry point is handled automatically, via mpi.h
Avoid large stack data allocation because stack size in AMPI is fixed and doesn’t grow at run time.
Can be specified with command line option.
9/17/2018
CS420

28. Outline Future work MPI basics Charm++/AMPI introduction
How to write AMPI programs
Running with virtualization
How to convert an MPI program
Using AMPI extensions
Automatic load balancing
Non-blocking collectives
Checkpoint/restart mechanism
Interoperability with Charm++
ELF and global variables
Future work
9/17/2018
CS420

29. AMPI Extensions Automatic load balancing Non-blocking collectives
Checkpoint/restart mechanism
Multi-module programming
ELF and global variables
9/17/2018
CS420

30. Automatic Load Balancing
2018/9/17
Automatic Load Balancing
Load imbalance in dynamic applications hurts the performance
Automatic load balancing: MPI_Migrate()
Collective call informing the load balancer that the thread is ready to be migrated, if needed.
If there is a load balancer present:
First sizing, then packing on source processor
Sending stack and packed data to the destination
Unpacking data on destination processor
Informs the load balancing system that you are ready to be migrated, if needed. If the system decides
to migrate you, the pup function passed to TCharm Register will first be called with a sizing pupper, then a
packing, deleting pupper. Your stack and pupped data will then be sent to the destination machine, where
your pup function will be called with an unpacking pupper. TCharm Migrate will then return. Can only be
called from in the parallel context.
9/17/2018
CS420

31. Automatic Load Balancing
2018/9/17
Automatic Load Balancing
To use automatic load balancing module:
Link with Charm’s LB modules
> charmc –o pgm hello.o -language ampi -module EveryLB
Run with +balancer option
> charmrun pgm +p4 +vp16 +balancer GreedyCommLB
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 200PE -> 50MB per PE
9/17/2018
CS420

32. Automatic Load Balancing
2018/9/17
Automatic Load Balancing
Link-time flag -memory isomalloc makes heap-data migration transparent
Special memory allocation mode, giving allocated memory the same virtual address on all processors
Ideal on 64-bit machines
Should fit in most cases and highly recommended
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 200PE -> 50MB per PE
9/17/2018
CS420

33. Automatic Load Balancing
2018/9/17
Automatic Load Balancing
Limitation with isomalloc:
Memory waste
4KB minimum granularity
Avoid small allocations
Limited space on 32-bit machine
Alternative: PUPer
Manually Pack/UnPack migrating data
(see the AMPI manual for PUPer examples)
Isomalloc: special memory allocation mode, give allocated memory the same virtual address on all processors.
problem: address space limitation on 32-bit machines. 4GB – OS, conventional heap allocation, = virtual address ~1GB
divided over 200PE -> 50MB per PE
9/17/2018
CS420

34. Automatic Load Balancing
Group your global variables into a data structure
Pack/UnPack routine (a.k.a. PUPer)
heap data –(Pack)–>
network message
–(Unpack)–> heap data
Demo – Load balancing
9/17/2018
CS420

35. Collective Operations
2018/9/17
Collective Operations
Problem with collective operations
Complex: involving many processors
Time consuming: designed as blocking calls in MPI
Powerful idea used in Charm++, now applied to AMPI too.
First this idea of non-blocking collectives was implemented in IBM MPI, but as AMPI is based on threads and capable of migration, it has more flexibility to take advantage of the overlapping feature.
Mention specially designed 2d fft.
“but is that possible?”
Time breakdown of 2D FFT benchmark [ms]
(Computation is a small proportion of elapsed time)
9/17/2018
CS420

36. Motivation for Collective Communication Optimization
2018/9/17
Motivation for Collective Communication Optimization
Powerful idea used in Charm++, now applied to AMPI too.
First this idea of non-blocking collectives was implemented in IBM MPI, but as AMPI is based on threads and capable of migration, it has more flexibility to take advantage of the overlapping feature.
Time breakdown of an all-to-all operation using Mesh library
Computation is only a small proportion of the elapsed time
A number of optimization techniques are developed to improve collective communication performance
9/17/2018
CS420

37. Asynchronous Collectives
Our implementation is asynchronous
Collective operation posted
test/wait for its completion
Meanwhile useful computation can utilize CPU
MPI_Ialltoall( … , &req);
/* other computation */
MPI_Wait(req);
9/17/2018
CS420

38. Asynchronous Collectives
2018/9/17
Asynchronous Collectives
Time breakdown of 2D FFT benchmark [ms]
VPs implemented as threads
Overlapping computation with waiting time of collective operations
Total completion time reduced
DON’T SAY the amount of reduced is equal to the amount of overlapped
9/17/2018
CS420

39. Checkpoint/Restart Mechanism
Large scale machines suffer from failure
Checkpoint/restart mechanism
State of applications checkpointed to disk files
Capable of restarting on different # of PE’s
Facilitates future efforts on fault tolerance
9/17/2018
CS420

40. Checkpoint/Restart Mechanism
Checkpoint with collective call
In-disk: MPI_Checkpoint(DIRNAME)
In-memory: MPI_MemCheckpoint(void)
Synchronous checkpoint
Restart with run-time option
In-disk: > ./charmrun pgm +p4 +restart DIRNAME
In-memory: automatic failure detection and resurrection
Demo: checkpoint/restart an AMPI program
9/17/2018
CS420

41. Interoperability with Charm++
Charm++ has a collection of support libraries
We can make use of them by running Charm++ code in AMPI programs
Also we can run MPI code in Charm++ programs
Demo: interoperability with Charm++
9/17/2018
CS420

42. ELF and global variables
Global variables are not thread-safe
Can we switch global variables when we switch threads?
The Executable and Linking Format (ELF)
Executable has a Global Offset Table containing global data
GOT pointer stored at %ebx register
Switch this pointer when switching between threads
Support on Linux, Solaris 2.x, and more
Integrated in Charm++/AMPI
Invoked by compile time option -swapglobals
Demo: thread-safe global variables
9/17/2018
CS420

43. Performance Visualization
Projections for AMPI
Register your function calls (e.g. foo)
REGISTER_FUNCTION(“foo”);
Replace your function calls you choose to trace with a macro
foo(10, “hello”); 
TRACEFUNC(foo(10, “hello”), “foo”);
Your function will be instrumented as a Projections event
9/17/2018
CS420

44. Outline Future work MPI basics Charm++/AMPI introduction
How to write AMPI programs
Running with virtualization
How to convert an MPI program
Using AMPI extensions
Automatic load balancing
Non-blocking collectives
Checkpoint/restart mechanism
Interoperability with Charm++
ELF and global variables
Future work
9/17/2018
CS420

45. Future Work Analyzing use of ROSE for application code manipulation
Improved support for visualization
Facilitating debugging and performance tuning
Support for MPI-2 standard
Complete MPI-2 features
Optimize one-sided communication performance
9/17/2018
CS420

46. Thank You!
Free download and manual available at:
Parallel Programming Lab at University of Illinois
9/17/2018
CS420