When do I need to use MPI_Barrier()?

Question

I wonder when do I need to use barrier? Do I need it before/after a scatter/gather for example? Or should OMPI ensure all processes have reached that point before scatter/gather-ing? Similarly, after a broadcast can I expect all processes to already receive the message?

Spiritualist

Answer 1

All collective operations in MPI before MPI-3.0 are blocking, which means that it is safe to use all buffers passed to them after they return. In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!) So MPI_Barrier is not necessary (or very helpful) before/after collective operations, if all buffers are valid already.

Please also note, that MPI_Barrier does not magically wait for non-blocking calls. If you use a non-blocking send/recv and both processes wait at an MPI_Barrier after the send/recv pair, it is not guaranteed that the processes sent/received all data after the MPI_Barrier. Use MPI_Wait (and friends) instead. So the following piece of code contains errors:

/* ERRORNOUS CODE */

Code for Process 0:
Process 0 sends something using MPI_Isend
MPI_Barrier(MPI_COMM_WORLD);
Process 0 uses buffer passed to MPI_Isend // (!)

Code for Process 1:
Process 1 recvs something using MPI_Irecv
MPI_Barrier(MPI_COMM_WORLD);
Process 1 uses buffer passed to MPI_Irecv // (!)

Both lines that are marked with (!) are unsafe!

MPI_Barrier is only useful in a handful of cases. Most of the time you do not care whether your processes sync up. Better read about blocking and non-blocking calls!

Kleist

Why is the 1st (!) an error? Process 0 will still have its own buffer? Also since its a send, the receiving party will not change it right? – Spiritualist

@JiewMeng MPI must not read from the buffer immediately after you call MPI_Isend. If you change it at (!), you might send something different. I am not quite sure about it, but I think that behaviour is undefined in this case. – Kleist

I've updated slightly your answer as MPI-3.0 introduced non-blocking collectives. – Planography

"In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!)" - isn't it inconsistent? How can all data be received without being sent? Maybe you've meant that because all collective operations are blocking, it's safe to reuse a buffer with the data-to-sent after a send call (because that's what "blocking" is about), because it's "copied" by MPI (not necessarily in the same way as for buffered send MPI_Bsend)? Of course it's correct that when blocking send returns we can't be sure that the data was received. – Pulsifer

And to make it clear... although with blocking send we can't be sure that the data was received, if we've used synchronized blocking send then when send returns we are certain that the recipient has received our message. – Pulsifer

@Pulsifer You are right, that's what I am saying. In my opinion the wording is not inconsistent, but I hope your comments improve clarity for people who feel the same way like you. Thank you! Just to repeat this once more: A blocking send does not imply that the message was sent and received, just that you can reuse buffers. A blocking receive call implies that all data was received. – Kleist

@MarkusMayr Hi, MarkusMayr, Your saying still confused me after I read it several times... What does "However, it does not imply that all data was sent!" mean?? Could you please explain it more clearly? – Uniat

@user15964: Your MPI implementation may create a copy of the buffers you passed to the blocking call. It may use that copy to send the data at a later point in time. For a collective operation, it is possible that your process has not finished sending data to other processes, but it already received all data that it needs, when the blocking call returns. – Kleist

@MarkusMayr That is much clearer, I think I understand your point now. Thank you very much! – Uniat

Thank you for the answer. Could you throw some light on whether MPI_Barrier is needed after calling topology functions such as MPI_Cart_Create (and before further processing of it with MPI_Cart_Shift or with MPI_Scatterv/Gatherv for example)? – Dismantle

Answer 2

One use of MPI_Barrier is for example to control access to an external resource such as the filesystem, which is not accessed using MPI. For example, if you want each process to write stuff to a file in sequence, you could do it like this:

int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
for ( int ii = 0; ii < size; ++ii ) {
    if ( rank == ii ) {
        // my turn to write to the file
        writeStuffToTheFile();
    }
    MPI_Barrier(MPI_COMM_WORLD);
}

That way, you can be sure that no two processes are concurrently calling writeStuffToTheFile.

Coccus

this is great for illustrative purpose. For this particular use case though, I'm wondering if there's some better, more efficient way? – Zicarelli

Yes, there is a whole range of MPI-IO support available for parallel file access if you are doing something more substantial. – Coccus

Answer 3

May MPI_Barrier() is not often used, but it is useful. In fact, even if you were use the synchronous communication, the MPI_Send/Recv() can only make sure the two processes is synchronized. In my project, a cuda+MPI project, all i used is asynchronous communication. I found that in some cases if i dont use the MPI_Barrier() followed by the Wait() function, the situation that two processes(gpu) want to transmit data to each other at the same time is very likely to happen, which could badly reduce the program efficiency. The bug above ever divers me mad and take me a few days to find it. Therefore you may think carefully whether use the MPI_Barrier() when you used the MPI_Isend/Irecv in your program. Sometimes sync the processes is not only neccessary but also MUST, especially ur program is dealing with the device.

Suppressive