Send Buffer Access MPI Forum meeting 1/2008

Send Buffer Access Background: MPI 1.1 standard prohibits users from accessing the send buffer for read until the send operation completes. Be it access in the same thread in case of an async send operation or by another thread in the case of blocking send operation. The rational in the MPI 1.1 standard was to enable the performance for DMA engine that is not cache-coherent with the main processor. Proposal: Remove the access restriction on the send buffers.

Why not? It is a change, and as a change to the standard requires a good reason Requires changing MPI impl. that do modify the buffer in place – Currently no known impl. This limitation affects performance – Currently no known impl. takes advantage of this restriction. Potentially future HW might take advantage of this restriction. For example, HW, that switches off the pages while sending.

Myth & old discussions Myth: computers with non-cache coherent design. (using multiple threads) – We learned that its not true; this can happen with a cache coherent machines too. or the non-cache coherent need explicitly to sync the caches. Old discussion: old mail thread on mpi-forum.org – Related to MPI impl on SGI machines that did not want to for overlapping memory regions. Mail thread with William C. Saphir from NASA re SGI machine performance. Okay with read from send buffer was not okay with 2 isends.

Why yes? Developers are surprised by this restriction – And/or writing non-portable code unknowingly Overlapped send perf: MPI_Isend(buf, … rank=3) MPI_Isend(buf, … rank=4) – Is illegal in MPI 2.0 – Perf implication: need to copy the buffer – Solution: use comm and bcast: too expensive – Send to a list of ranks: API does not exist (yet?) Overlapped computation perf: – One thread MPI_Isend & compute pattern (using same memory) Multi threaded computation perf: – One thread, in a collective operation – Second thread use same memory for computation

Work group list Erez Haba David Gingold Gil Bloch George Bosilca Darius Buntinas Dries Kimpe Patrick Geoffray Doug Gregor