Ok to call MPI_Isend multiple times on one buffer?

Question

Regarding MPI_Isend, the MPI standard says "A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not access any part of the send buffer after a nonblocking send operation is called, until the send completes." (http://www.mpi-forum.org/docs/mpi-11-html/node46.html)

Is referencing the send buffer in another send call ok, or is that included in "access any part of the send buffer"?

In other words, is the following C code for the sender correct?

MPI_Request req[2];
MPI_Status statuses[2];
...
MPI_Isend(buf, type, count, dest0, tag, comm, &req[0]);
MPI_Isend(buf, type, count, dest1, tag, comm, &req[1]);
MPI_Waitall(2, req, statuses);

Lovell

I'm guessing here, which is why this is not going in as a full answer, that what you really don't want to do is modify the contents of the source buffer or read the contents of the destination buffer before the send is complete. So to me, what you are doing looks OK. But please test. Don't take my word for it. – Madame

@Madame Of course, that's true. And I know my code would be correct for the MPI implementation I'm using (looked at the source code), but I'd like to know if this is correct according to the Standard, i.e. would work on every correct implementation of MPI. – Lovell

Answer 1

The MPI Standard does allow for this kind of usage.

If there are more that a "handful" of ranks that need the same buffer, or if this communication patter will be repeated more than a "handful" of times...then creating a comm with the relevant ranks, and using MPI_Bcast would be preferable.

EDIT:

To clarify my own answer. The MPI 2.0 Standard specifically prohibited this kind of usage. The restriction was to accommodate Fortran. The MPI 2.1 or 2.2 Standard included a "clarification" that this re-use of a send buffer in multiple ISends was permissible. See Section 16.2.2 of the MPI 2.2 Standard for more.

Calle

Thanks. That section of MPI 2.2 Standard was enlightening. – Lovell

Answer 2

I just consulted the MPI 3.0 Standard and found the following information:

A nonblocking send call indicates that the system may start copying data out of the send buffer. The sender should not modify any part of the send buffer after a nonblocking send operation is called, until the send completes.

However, I very recently attended a tutorial on MPI 3.0 given by some of the developers of the standard and it was mentioned that, depending on the MPI implementation (MPICH, LAM, etc.), it may be unsafe to access a buffer used in a nonblocking communication call as the buffer may be modified from the Isend procedure before the send is complete. In other words, it might not be guaranteed that the buffer in the second MPI_Isend is the same as the buffer in the first send. In order to figure this out for sure, I would consult your implementation source code.

Ingurgitate

Hmmmm, this is what I was afraid of. Does the Standard specify that send buffers at the time of send completion must be "returned" to the application unmodified, or that they may be modified? I just cannot see how a reasonable implementation would benefit by actually modifying a send buffer. – Lovell

It's not that the implementation would necessarily modify a buffer, but it does need to ensure that the buffer isn't changed while it's being copied/sent. – Stalder