An Introduction to Parallel Programming with MPI March 22, 24, 29, 31 2005 David Adams daadams3@vt.edu http://research.cs.vt.edu/lasca/schedule

Outline Collective communications Disclaimers Overview of basic parallel programming on a cluster with the goals of MPI Batch system interaction Startup procedures Quick review Blocking message passing Non-blocking message passing Collective communications

Review Functions we have covered in detail: Useful constants: MPI_INIT MPI_FINALIZE MPI_COMM_SIZE MPI_COMM_RANK MPI_SEND MPI_RECV MPI_ISEND MPI_IRECV MPI_WAIT MPI_TEST Useful constants: MPI_COMM_WORLD MPI_ANY_SOURCE MPI_ANY_TAG MPI_SUCCESS MPI_REQUEST_NULL MPI_TAG_UB

Collective Communications Transmit data to all processes within a communicator domain. (All processes in MPI_COMM_WORLD for example.) Called by every member of a communicator but can not be relied on to synchronize the processes (except MPI_BARRIER). Come only in blocking versions and standard mode semantics. Collective communications are SLOW but are a convenient way of passing the optimization of data transfer to the vendor instead of the end user. Everything accomplished with collective communications could also be done using the functions we have already gone over. They are simply shortcuts and implementer optimizations for communication patterns that are used often by parallel programmers.

BARRIER MPI_BARRIER(COMM, IERROR) IN INTEGER COMM OUT IERROR Blocks the caller until all processes in the group have entered the call to MPI_BARRIER. Allows for process synchronization and is the only collective operation that guarantees synchronization at the call even though others could synchronize as a side effect.

Broadcast MPI_BCAST(BUFFER, COUNT, DATATYPE, ROOT, COMM, IERROR) INOUT <type> BUFFER(*) IN INTEGER COUNT, DATATYPE, ROOT, COMM OUT IERROR Broadcasts a message from the process with rank root to all processes of the communicator group. Serves as both the blocking send and blocking receive for message completion and must be called by every processor in the communicator group. Conceptually, this can be viewed as sending a single message from root to every processor in the group but MPI implementations are free to make this more efficient. On return, the contents of the root processor’s BUFFER have been copied to all processes

Broadcast Data  Data  A0 A0  Processes  Processes A0 A0 A0 A0 A0

Gather MPI_GATHER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE, COMM, IERROR) OUT <type> RECVBUF(*) IN <type> SENDBUF(*) IN INTEGER SENDCOUNT, RECVCOUNT, SENDTYPE, RECVTYPE, COMM OUT IERROR Each process (including the root) sends the contents of its send buffer to the root process. The root process collects the messages in rank order and stores them in the RECVBUF. If there are n processes in the communicator group then the RECVBUF must be n times larger than the SENDBUF. RECVCOUNT = SENDCOUNT, meaning that the function is looking for the count of objects of type RECVTYPE that it is receiving from each process.

Gather Data  Data  A0 A0 A1 A2 A3 A4 A5  Processes A1  Processes

Scatter MPI_SCATTER(SENDBUF, SENDCOUNT, SENDTYPE, RECVBUF, RECVCOUNT, RECVTYPE, COMM, IERROR) OUT <type> RECVBUF(*) IN <type> SENDBUF(*) IN INTEGER SENDCOUNT, RECVCOUNT, SENDTYPE, RECVTYPE, COMM OUT IERROR MPI_SCATTER is the inverse of MPI_GATHER. The outcome of this function is for root to take its SENDBUF and split it into n equal segments, 0 through (n-1), where the ith segment is delivered to the ith process in the group.

Scatter Data  Data  A0 A1 A2 A3 A4 A5 A0  Processes  Processes A1

ALLGATHER Data  Data  A0 A0 B0 C0 D0 E0 F0  Processes B0

ALLTOALL Data  Data  A0 A1 A2 A3 A4 A5 A0 B0 C0 D0 E0 F0  Processes

Global Reductions MPI can perform a global reduction operation across all members of a communicator group. Reduction operations include operations like: Maximum Minimum Sum Product ANDs and ORs

MPI_REDUCE MPI_REDUCE(SENDBUF, RECVBUF, COUNT, DATATYPE, OP, ROOT, COMM, IERROR) OUT <type> RECVBUF(*) IN <type> SENDBUF(*) IN INTEGER COUNT, DATATYPE, OP, ROOT, COMM OUT IERROR Combines the elements provided in the input buffer of each process in the group, using the operation OP, and returns the combined value in the output buffer of the process with rank ROOT. Predefined operations include: MPI_MAX MPI_MIN MPI_SUM MPI_PROD MPI_LAND MPI_BAND MPI_LOR MPI_BOR MPI_LXOR MPI_BXOR

Helpful Online Information Man pages for MPI: http://www-unix.mcs.anl.gov/mpi/www/ MPI homepage at Argonne National Lab: http://www-unix.mcs.anl.gov/mpi/ Some more sample programs: http://www-unix.mcs.anl.gov/mpi/usingmpi/examples/main.htm Other helpful books: http://fawlty.cs.usfca.edu/mpi/ http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=3614 Some helpful UNIX commands: http://www.ee.surrey.ac.uk/Teaching/Unix/