Immidiate vs synchronous communication in openmpi

Question

I got slightly mixed up regarding the concept of synchronous - asynchronous in the context of blocking & non blocking operations (in OpenMPI) from here:

• link 1 : MPI_Isend is not necessarily asynchronous ( so it can synchronous ?)

• link 2 :The MPI_Isend() and MPI_Irecv() are the ASYNCHRONOUS communication primitives of MPI.

I have already gone through the previous sync - async - blocking - non blocking questions on stackoverflow (asynchronous vs non-blocking), but were of no help to me. As far as i know :

• Immediate (MPI_Isend): method returns & executes next line -> nonblocking
• Standard/non-immediate (MPI_Send) : for large messages it blocks until transfer completes
• A synchronous operation blocks (http://www.cs.unc.edu/~dewan/242/s07/notes/ipc/node9.html)
• An asynchronous operation is non-blocking (http://www.cs.unc.edu/~dewan/242/s07/notes/ipc/node9.html)

so how & why MPI_ISEND may be blocking (link 1) as well as non blocking (link 2) ? i.e.what is meant by asynchronous & synchronous MPI_Isend here ?

Similar confusion arises regarding MPI_Ssend & MPI_Issend, since the S in MPI_SSEND means synchronous(or blocking) and:-

• MPI_Ssend: synchronous send blocks until data is received on remote process & ack is received by sender,
• MPI_Issend: means immediate synchronous send

also the Immediate is non-blocking, So, how can MPI_ISSEND be Synchronous & return Immediately ?

I guess more clarity is needed in asynchronous & synchronous in context of blocking & non blocking OpenMPI communication . A practical example or analogy in this regard will be very useful.

Haskins

I was trying to understand the same thing, and I ended up writing examples for each send mode, here: gist.github.com/deborasetton/9f8597d85a4d314ace17. Sometimes it's useful to look at actual code to see the differences. – Rule

Answer 1

There is a distinction between when MPI function calls return (blocking vs. non-blocking) and when the corresponding operation completes (standard, synchronous, buffered, ready-mode).

Non-blocking calls MPI_I... return immediately, no matter if the operation has completed or not. The operation continues in the background, or asynchronously. Blocking calls do not return unless the operation has completed. Non-blocking operations are represented by their handle which could be used to perform a blocking wait (MPI_WAIT) or non-blocking test (MPI_TEST) for completion.

Completion of an operation means that the supplied data buffer is no longer being accessed by MPI and it could therefore be reused. Send buffers become free for reuse either after the message has been put in its entirety on the network (including the case where part of the message might still be buffered by the network equipment and/or the driver), or has been buffered somewhere by the MPI implementation. The buffered case does not require that the receiver has posted a matching receive operation and hence is not synchronising - the receive could be posted at a much later time. The blocking synchronous send MPI_SSEND does not return unless the receiver has posted a receive operation, thus it synchronises both ranks. The non-blocking synchronous send MPI_ISSEND returns immediately but the asynchronous (background) operation won't complete unless the receiver has posted a matching receive.

A blocking operation is equivalent to a non-blocking one immediately followed by a wait. For example:

MPI_Ssend(buf, len, MPI_TYPE, dest, tag, MPI_COMM_WORLD);

is equivalent to:

MPI_Request req;
MPI_Status status;

MPI_Issend(buf, len, MPI_TYPE, dest, tag, MPI_COMM_WORLD, &req);
MPI_Wait(&req, &status);

The standard send (MPI_SEND / MPI_ISEND) completes once the message has been constructed and the data buffer provided as its first argument might be reused. There is no synchronisation guarantee - the message might get buffered locally or remotely. With most implementations, there is usually some size threshold: messages up to that size get buffered while longer messages are sent synchronously. The threshold is implementation-dependent.

Buffered sends always buffer the messages into a user-supplied intermediate buffer, essentially performing a more complex memory copy operation. The difference between the blocking (MPI_BSEND) and the non-blocking version (MPI_IBSEND) is that the former does not return before all the message data has been buffered.

The ready send is a very special kind of operation. It only completes successfully if the destination rank has already posted the receive operation by the time when the sender makes the send call. It might reduce the communication latency by eliminating the need to performing some kind of a handshake.

Vocation

As a follow up, what does MPI_Send since you've already explained MPI_SSend and MPI_ISend? – Postern

MPI_Send is the so-called standard send operation - it either buffers the message or sends it in the same way MPI_Ssend does. Unlike MPI_Ssend, it doesn't guarantee that the receiver has started receiving the message, only that the message has entered the MPI library and the data buffer can be further reused. – Vocation