1. An Introduction to Parallel Programming with MPI
March 22, 24, 29, 31
2005
David Adams

2. Outline Collective communications Disclaimers
Overview of basic parallel programming on a cluster with the goals of MPI
Batch system interaction
Startup procedures
Quick review
Blocking message passing
Non-blocking message passing
Collective communications

3. Review Functions we have covered in detail: Useful constants:
MPI_INIT MPI_FINALIZE
MPI_COMM_SIZE MPI_COMM_RANK
MPI_SEND MPI_RECV
MPI_ISEND MPI_IRECV
MPI_WAIT MPI_TEST
Useful constants:
MPI_COMM_WORLD MPI_ANY_SOURCE
MPI_ANY_TAG MPI_SUCCESS
MPI_REQUEST_NULL MPI_TAG_UB

4. Collective Communications
Transmit data to all processes within a communicator domain. (All processes in MPI_COMM_WORLD for example.)
Called by every member of a communicator but can not be relied on to synchronize the processes (except MPI_BARRIER).
Come only in blocking versions and standard mode semantics.
Collective communications are SLOW but are a convenient way of passing the optimization of data transfer to the vendor instead of the end user.
Everything accomplished with collective communications could also be done using the functions we have already gone over. They are simply shortcuts and implementer optimizations for communication patterns that are used often by parallel programmers.

5. BARRIER MPI_BARRIER(COMM, IERROR)
IN INTEGER COMM
OUT IERROR
Blocks the caller until all processes in the group have entered the call to MPI_BARRIER.
Allows for process synchronization and is the only collective operation that guarantees synchronization at the call even though others could synchronize as a side effect.

6. Broadcast MPI_BCAST(BUFFER, COUNT, DATATYPE, ROOT, COMM, IERROR)
INOUT <type> BUFFER(*)
IN INTEGER COUNT, DATATYPE, ROOT, COMM
OUT IERROR
Broadcasts a message from the process with rank root to all processes of the communicator group.
Serves as both the blocking send and blocking receive for message completion and must be called by every processor in the communicator group.
Conceptually, this can be viewed as sending a single message from root to every processor in the group but MPI implementations are free to make this more efficient.
On return, the contents of the root processor’s BUFFER have been copied to all processes

7. Broadcast
Data 
Data  A0 A0
 Processes
 Processes A0 A0 A0 A0 A0

8. Gather
MPI_GATHER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE, COMM, IERROR)
OUT <type> RECVBUF(*)
IN <type> SENDBUF(*)
IN INTEGER SENDCOUNT, RECVCOUNT, SENDTYPE, RECVTYPE, COMM
OUT IERROR
Each process (including the root) sends the contents of its send buffer to the root process.
The root process collects the messages in rank order and stores them in the RECVBUF.
If there are n processes in the communicator group then the RECVBUF must be n times larger than the SENDBUF.
RECVCOUNT = SENDCOUNT, meaning that the function is looking for the count of objects of type RECVTYPE that it is receiving from each process.

9. Gather Data  Data  A0 A0 A1 A2 A3 A4 A5  Processes A1  Processes

10. Scatter
MPI_SCATTER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE, COMM, IERROR)
OUT <type> RECVBUF(*)
IN <type> SENDBUF(*)
IN INTEGER SENDCOUNT, RECVCOUNT, SENDTYPE, RECVTYPE, COMM
OUT IERROR
MPI_SCATTER is the inverse of MPI_GATHER.
The outcome of this function is for root to take its SENDBUF and split it into n equal segments, 0 through (n-1), where the ith segment is delivered to the ith process in the group.

11. Scatter Data  Data  A0 A1 A2 A3 A4 A5 A0  Processes  Processes A1

12. ALLGATHER Data  Data  A0 A0 B0 C0 D0 E0 F0  Processes B0

13. ALLTOALL Data  Data  A0 A1 A2 A3 A4 A5 A0 B0 C0 D0 E0 F0  Processes

14. Global Reductions
MPI can perform a global reduction operation across all members of a communicator group.
Reduction operations include operations like:
Maximum
Minimum
Sum
Product
ANDs and ORs

15. MPI_REDUCE
MPI_REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM, IERROR)
OUT <type> RECVBUF(*)
IN <type> SENDBUF(*)
IN INTEGER COUNT, DATATYPE, OP, ROOT, COMM
OUT IERROR
Combines the elements provided in the input buffer of each process in the group, using the operation OP, and returns the combined value in the output buffer of the process with rank ROOT.
Predefined operations include:
MPI_MAX MPI_MIN MPI_SUM
MPI_PROD MPI_LAND MPI_BAND
MPI_LOR MPI_BOR MPI_LXOR
MPI_BXOR

16. Helpful Online Information
Man pages for MPI:
MPI homepage at Argonne National Lab:
Some more sample programs:
Other helpful books:
Some helpful UNIX commands: