Parallel Programming with MPI Prof. Sivarama Dandamudi School of Computer Science Carleton University

© S. Dandamudi2 Introduction  Problem  Lack of a standard for message–passing routines  Big issue: Portability problems  MPI defines a core set of library routines (API) for message passing (more than 125 functions in total!!)  Several commercial and public domain implementations  Cray, IBM, Intel  MPI-CH from Argonne National Laboratory  LAM from Ohio Supercomputer Center/Indiana University

Carleton University© S. Dandamudi3 Introduction (cont’d)  Some Additional Goals [Snir et al. 1996]  Allows efficient communication  Avoids memory-to-memory copying  Allows computation and communication overlap  Non-blocking communication  Allows implementation in a heterogeneous environment  Provides reliable communication interface  Users don’t have to worry about communication failures

Carleton University© S. Dandamudi4 MPI  MPI is large but not complex  125 functions  But….  Need only 6 functions to write a simple MPI program  MPI_Init  MPI_Finalize  MPI_Comm_size  MPI_Comm_rank  MPI_Send  Mpi_Recv

Carleton University© S. Dandamudi5 MPI (cont’d)  Before any other function is called, we must initialize MPI_Init(&argc, &argc)  To indicate end of MPI calls  MPI_Finalize()  Cleans up the MPI state  Should be the last MPI function call

Carleton University© S. Dandamudi6 MPI (cont’d)  A typical program structure int main(int argc, char **argv) { MPI_Init(&argc, &argv);... /* main program */... MPI_Finalize(); }

Carleton University© S. Dandamudi7 MPI (cont’d)  MPI uses communicators to group processes that communicate with each other  Predefined communicator: MPI_COMM_WORLD consists of all processes running when the program begins execution  Sufficient for simple programs

Carleton University© S. Dandamudi8 MPI (cont’d)  Process rank  Similar to mytid in PVM MPI_Comm_rank(MPI_Comm comm, int *rank)  First argument: Communicator  Second argument: returns process rank

Carleton University© S. Dandamudi9 MPI (cont’d)  Number of processes MPI_Comm_size(MPI_Comm comm, int *size)  First argument: Communicator  Second argument: returns number of processes  Example: MPI_Comm_size(MPI_COMM_WORLD, &nprocs)

Carleton University© S. Dandamudi10 MPI (cont’d)  Sending a message (blocking version) MPI_Send(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm )  Data types  MPI_CHAR, MPI_INT, MPI_LONG  MPI_FLOAT, MPI_DOUBLE Buffer description Destination specification

Carleton University© S. Dandamudi11 MPI (cont’d)  Receiving a message (blocking version) MPI_Recv(void* buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status)  Wildcard specification allowed  MPI_ANY_SOURCE  MPI_ANY_TAG

Carleton University© S. Dandamudi12 MPI (cont’d)  Receiving a message  Status of the received message  Status gives two pieces of information directly  Useful when wildcards are used  status.MPI_SOURCE  Gives identity of the source  status.MPI_TAG  Gives the tag information

Carleton University© S. Dandamudi13 MPI (cont’d)  Receiving a message  Status also gives message size information indirectly MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, int *count)  Takes status and the datatype as inputs and returns the number of elements via count

Carleton University© S. Dandamudi14 MPI (cont’d)  Non-blocking communication  Prefix send and recv by “I” (for immediate)  MPI_Isend  MPI_Irecv  Need completion operations to see if the operation is completed  MPI_Wait  MPI_Test

Carleton University© S. Dandamudi15 MPI (cont’d)  Sending a message (non-blocking version) MPI_Isend(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request ) Returns the request handle

Carleton University© S. Dandamudi16 MPI (cont’d)  Receiving a message (non-blocking version) MPI_Irecv(void* buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request )  Same arguments as Isend

Carleton University© S. Dandamudi17 MPI (cont’d)  How do we know when a non-blocking operation is done?  Use MPI_Test or MPI_Wait  Completion of a send indicates:  Sender can access the send buffer  Completion of a receive indicates  Receive buffer contains the message

Carleton University© S. Dandamudi18 MPI (cont’d)  MPI_Test returns the status  Does not wait for the operation to complete MPI_Test(MPI_Request*request, int *flag, MPI_Status *status) Request handle Operation status: true (if completed) If flag = true, gives status

Carleton University© S. Dandamudi19 MPI (cont’d)  MPI_Wait waits until the operation is completed MPI_Wait( MPI_Request *request, MPI_Status *status) Request handle Gives status

Carleton University© S. Dandamudi20 MPI Collective Communication  Collective communication  Several functions are provided to support collective communication  Some examples are given here:  MPI_Barrier  MPI_Bcast  MPI_Scatter  MPI_Gather  MPI_Reduce Broadcast Barrier synchronization Global reduction

Carleton University© S. Dandamudi21 From Snir et al. 1996

Carleton University© S. Dandamudi22 MPI Collective Communication (cont’d)  MPI_Barrier blocks the caller until all group members have called it MPI_Barrier( MPI_Comm comm )  The call returns at any process only after all group members have entered the call

Carleton University© S. Dandamudi23 MPI Collective Communication (cont’d)  MPI_Bcast broadcasts a message from root to all processes of the group MPI_Bcast( void* buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm )

Carleton University© S. Dandamudi24 MPI Collective Communication (cont’d)  MPI_Scatter distributes data from the root process to all the others in the group MPI_Scatter(void* send_buf, int send_count, MPI_Datatype send_type, void* recv_buf, int recv_count, MPI_Datatype recv_type, int root, MPI_Comm comm )

Carleton University© S. Dandamudi25 MPI Collective Communication (cont’d)  MPI_Gather inverse of the scatter operation (gathers data and stores it in rank order) MPI_Scatter(void* send_buf, int send_count, MPI_Datatype send_type, void* recv_buf, int recv_count, MPI_Datatype recv_type, int root, MPI_Comm comm )

Carleton University© S. Dandamudi26 MPI Collective Communication (cont’d)  MPI_Reduce performs global reduction operations such as sum, max, min, AND, etc. MPI_Reduce(void* send_buf, void* recv_buf, int count, MPI_Datatype datatype, MPI_Op operation, int root, MPI_Comm comm )

Carleton University© S. Dandamudi27 MPI Collective Communication (cont’d)  Predefined reduce operations include MPI_MAXmaximum MPI_MINminimum MPI_SUMsum MPI_PRODproduct MPI_LANDlogical AND MPI_BANDbitwise AND MPI_LORlogical OR MPI_BORbitwise OR MPI_LXORlogical XOR MPI_BXORbitwise XOR

Carleton University© S. Dandamudi28 From Snir et al Last slide