1 Overview on Send And Receive routines in MPI Kamyar Miremadi November 2004

2 Function MPI_Send int MPI_Send ( void *message, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm )

3 Function MPI_Recv int MPI_Recv ( void *message, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status )

4 Coding Send/Receive … if (ID == j) { … Receive from I … } … if (ID == i) { … Send to j … } …

5 Send/Receive Not Collective

6 Inside MPI_Send and MPI_Recv Sending ProcessReceiving Process Program Memory System Buffer System Buffer Program Memory MPI_Send MPI_Recv

7 Inside MPI_Send and MPI_Recv Return from MPI_Send  Function blocks until message buffer free  Message buffer is free when Message copied to system buffer, or Message transmitted Return from MPI_Recv  Function blocks until message in buffer  If message never arrives, function never returns

8 Deadlock Deadlock: process waiting for a condition that will never become true Easy to write send/receive code that deadlocks  Two processes: both receive before send  Send tag doesn’t match receive tag  Process sends message to wrong destination process

9 Deadlock Example Will always deadlock, no matter the buffering mode. if(rank==0) { MPI_Recv(...); MPI_Send(...); } else if(rank==1) { MPI_Recv(...); MPI_Send(...); }

10 Send and Receive Synchronization Fully Synchronized (Rendezvous)  Send and Receive complete simultaneously whichever code reaches the Send/Receive first waits  provides synchronization point (up to network delays) Buffered  Receive must wait until message is received  Send completes when message is moved to buffer clearing memory of message for reuse Asynchronous  Sending process may proceed immediately does not need to wait until message is copied to buffer must check for completion before using message memory  Receiving process may proceed immediately will not have message to use until it is received must check for completion before using message

11 Communication modes MPI defines four communication modes  synchronous mode ("safest")  ready mode (lowest system overhead)  buffered mode (decouples sender from receiver)  standard mode (compromise) Communication mode is selected with send routine Calls are also blocking or nonblocking.  Blocking stops the program until the message buffer is safe to use  Non-blocking separates communication from computation

12 Communication modes Buffer-mode: send operation can be started whether or not a matching receive has been posted. It may complete before a matching receive is posted. synchronous-mode: send can be started whether or not a matching receive was posted. However, the send will complete successfully only if a matching receive is posted, and the receive operation has started to receive the message sent by the synchronous send. ready-mode send may be started only if the matching receive has already been posted. standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. A send in standard mode can be started whether or not a matching receive has been posted.

13 Communication modes(cont.) AdvantagesDisadvantages SynchronousSafest, and therefore most portable SEND/RECV order not critical Amount of buffer space irrelevant Can incur substantial syncronization overhead ReadyLowest total overhead SEND/RECV handshake not required RECV must precede SEND BufferedDecouples SEND from RECV No sync overhead on SEND Order of SEND/RECV irrelevant Programmer can control size of buffer space Additional system overhead incurred by copy to buffer Standard Good for many cases Your program may not be suitable

14 Blocking and Non-Blocking Send and receive can be blocking or non- blocking A blocking send can be used with a non- blocking receive, and vice-versa Non-blocking sends can use any mode -- synchronous, buffered, standard, or ready

15 Communication modes(cont.) Communication Blocking Non-Blocking Mode Routines Routines Synchronous MPI_SSEND MPI_ISSEND Ready MPI_RSEND MPI_IRSEND Buffered MPI_BSEND MPI_IBSEND Standard MPI_SEND MPI_ISEND MPI_RECV MPI_IRECV

16 Completion Tests Waiting vs. Testing int MPI_Wait(MPI_Request *request,MPI_Status *status) int MPI_Test(MPI_Request *request,int *flag,MPI_Status *status)

17 Comparisons & General Use Blocking: call MPI_RECV (x,N,MPI_Datatype,…,status,…) Non-Blocking: call MPI_IRECV (x,N,MPI_Datatype,…,request,…) … do work that does not involve array x call MPI_WAIT (request,status) … do work that does involve array x Non-Blocking: call MPI_IRECV (x,N,MPI_Datatype,…,request,…) call MPI_TEST (request,flag,status,…) do while (flag.eq. FALSE) … work that does not involve the array x … call MPI_TEST (request,flag,status,…) end do … do work that does involve the array x …

18 Questions?