Is MPI_Bcast() blocking?

Question

Is MPI_Bcast() blocking or nonblocking? In other word, when the root sends a data, do all processors block until every processor has received this data? If not, how to synchronized (block) all of them so that no one proceeds until all receives the same data.

Barghest

The only MPI collective communication call guaranteed to be synchronising is MPI_Barrier. – Christiano

Answer 1

No, this kind of blocking (waiting for the other processes to finish their part of the job) would be very bad for performance. Every process continues as soon as it has all it need -- that means that the data it was to receive are there, or the data to be sent are at least copied to some buffer.

You can use an MPI_Barrier to synchronize processes if you need to be sure all processes finished. As already said, it can slowdown the program significantly. I use it only for certain diagnostic logging when initializing my code. Not during the actual integration.

Kristiekristien

Answer 2

You need to be a bit careful about terminology here as what MPI means by "blocking" may not be how you have seen it used in other contexts.

In MPI terms, Bcast is blocking. Blocking means that, when the function returns, it has completed the operation it was meant to do. In this case, it means that on return from Bcast it is guaranteed that the receive buffer in every process contains the data you want to broadcast. The non-blocking version is Ibcast.

In MPI terms, what you are asking is whether the operation is synchronous, i.e. implies synchronisation amongst processes. For a point-to-point operation such as Send, this refers to whether or not the sender waits for the receive to be posted before returning from the send call. For collective operations, the question is whether there is a barrier (as pointed out by @Vladimir). Bcast does not necessarily imply a barrier.

However, the reason I am posting is that, in almost all MPI programs written using the standard Send/Recv calls (as opposed to single-sided Put/Get) you do not care if there is a synchronisation after the barrier. All each process cares about is that it has received the data it needs - why would it matter what the other processes are doing? If you subsequently want to communicate with any other process then the MPI routines are designed so that the required synchronisation happens automatically. If you issue a receive and another process is slow, you wait; if you issue a send and the other process has not issued a receive, everything will still work correctly (this is assuming you don't call Rsend - you should never call Rsend!). Whether or not there is synchronisation has effects on performance, but rarely affects whether a program is correct or not.

Unless processes are interacting via some other mechanism (e.g. all accessing the same file) then it is hard to come up with a real example where you care whether or not the Bcast synchronises. Of course you can always construct some edge case, but in real practical applications of MPI it almost never matters.

Many MPI programs are littered with barriers and in my experience they are almost never required for correctness; the only common use case is to ensure meaningful timings for performance measurements.

Sable

"... it means that on return from Bcast it is guaranteed that the receive buffer in every process contains the data you want to broadcast." - This is not true for the root rank. Blocking operations return once the local part of the operation has completed and the user-supplied buffer can be reused. In the case of broadcast, the operation could complete at the root while there are still messages in transit. I fully agree with the rest of your answer, especially with the last paragraph. – Christiano

OK - you're correct my answer was not quite clear enough. What I meant to say was that "it means that when a process returns from Bcast it is guaranteed that the process's receive buffer contains the data you want to broadcast". You're right that my original text incorrectly implied that this was true globally rather than just locally. – Sable

MPI_Barriers are also useful to assure that a given step in the initialization has fully completed so that one can trust the diagnostic log one writes. Identifying the location where a problem occurred is much faster. It is executed only once anyway, so the slowdown does not matter too much. – Rarefied

Another case. We were just bitten by a bug in a particular HPC MPI implementation where MPI_Scatter or a similar collective will cease to work after 10-15 minutes of waiting. An MPI_Barrier will cure this. Removing an MPI_Barrier just because it is not strictly necessary can have very unpredictable results in practical large computations. – Rarefied