thread safety of MPI send using threads created with std::async

Question

According to this website, the usage of MPI::COMM_WORLD.Send(...) is thread safe. However in my application I often (not always) run into deadlocks or get segmentation faults. Enclosing each call of MPI::COMM_WORLD methods with a mutex.lock() and mutex.unlock() consistently removes deadlocks as well as segfaults.

This is how I create threads:

const auto communicator = std::make_shared<Communicator>();
std::vector<std::future<size_t>> handles;
for ( size_t i = 0; i < n; ++i )
{
   handles.push_back(std::async(std::launch::async, foo, communicator));
}
for ( size_t i = 0; i < n; ++i )
{
   handles[i].get();
}

Communicator is a class which has a std::mutex member and exclusively calls methods such as MPI::COMM_WORLD.Send() and MPI::COMM_WORLD.Recv(). I do not use any other methods of sending/receiving with MPI. foo takes a const std::shared_ptr<Commmunicator> & as argument.

My question: Is the thread safety promised by MPI not compatible with threads created by std::async?

Porush

Answer 1

Thread-safety in MPI doesn't work out of the box. First, you have to ensure that your implementation actually supports multiple threads making MPI calls at once. With some MPI implementations, for example Open MPI, this requires the library to be configured with special options at build time. Then you have to tell MPI to initialise at the appropriate thread support level. Currently the MPI standard defines four levels of thread support:

• MPI_THREAD_SINGLE - means that the user code is single threaded. This is the default level at which MPI is initialised if MPI_Init() is used;
• MPI_THREAD_FUNNELED - means that the user code is multithreaded, but only the main thread makes MPI calls. The main thread is the one which initialises the MPI library;
• MPI_THREAD_SERIALIZED - means that the user code is multithreaded, but calls to the MPI library are serialised;
• MPI_THREAD_MULTIPLE - means that the user code is multithreaded and all threads can make MPI calls at any time with no synchronisation whatsoever.

In order to initialise MPI with thread support, one has to use MPI_Init_thread() instead of MPI_Init():

int provided;

MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
if (provided < MPI_THREAD_MULTIPLE)
{
    printf("ERROR: The MPI library does not have full thread support\n");
    MPI_Abort(MPI_COMM_WORLD, 1);
}

Equivalent code with the obsoleted (and removed from MPI-3) C++ bindings:

int provided = MPI::Init_thread(argc, argv, MPI::THREAD_MULTIPLE);
if (provided < MPI::THREAD_MULTIPLE)
{
    printf("ERROR: The MPI library does not have full thread support\n");
    MPI::COMM_WORLD.Abort(1);
}

Thread support levels are ordered like this: MPI_THREAD_SINGLE < MPI_THREAD_FUNNELED < MPI_THREAD_SERIALIZED < MPI_THREAD_MULTIPLE, so any other provided level, different from MPI_THREAD_MULTIPLE would have lower numerical value - that's why the if (...) code above is written so.

MPI_Init(&argc, &argv) is equivalent to MPI_Init_thread(&argc, &argv, MPI_THREAD_SINGLE, &provided). Implementations are not required to initialise exactly at the requested level - rather they could initialise at any other level (higher or lower), which is returned in the provided output argument.

For more information - see §12.4 of the MPI standard, freely available here.

With most MPI implementations, the thread support at level MPI_THREAD_SINGLE is actually equivalent to that provided at level MPI_THREAD_SERIALIZED - exactly what you observe in your case.

Since you've not specified which MPI implementation you use, here comes a handy list.

I've already said that Open MPI has to be compiled with the proper flags enabled in order to support MPI_THREAD_MULTIPLE. But there is another catch - its InfiniBand component is not thread-safe and hence Open MPI would not use native InfiniBand communication when initialised at full thread support level.

Intel MPI comes in two different flavours - one with and one without support for full multithreading. Multithreaded support is enabled by passing the -mt_mpi option to the MPI compiler wrapper which enables linking with the MT version. This option is also implied if OpenMP support or the autoparalleliser is enabled. I am not aware how the InfiniBand driver in IMPI works when full thread support is enabled.

MPICH(2) does not support InfiniBand, hence it is thread-safe and probably most recent versions provide MPI_THREAD_MULTIPLE support out of the box.

MVAPICH is the basis on which Intel MPI is built and it supports InfiniBand. I have no idea how it behaves at full thread support level when used on a machine with InfiniBand.

The note about multithreaded InfiniBand support is important since lot of compute clusters nowadays use InfiniBand fabrics. With the IB component (openib BTL in Open MPI) disabled, most MPI implementations switch to another protocol, for example TCP/IP (tcp BTL in Open MPI), which results in much slower and more latent communication.

Mugger

Very helpful answer, thank you very much! Since I have to work with multiple different mpi implementations, I sadly only have the option to use my own mutex, but that's still a decent solution. – Porush

You always have the option to try to initialise MPI with full thread support and employ the mutex only if you don't get the requested thread support level. Note that being able to make serialised multithreaded MPI calls at MPI_THREAD_SINGLE or MPI_THREAD_FUNNELED is not a standard-specified behaviour. – Mugger

yes, this is indeed a possibility. However from a design perspective I wouldn't do that. It either involves conditionals in the whole communication code or the conditional creation of the communicator object. I dislike that, so I decided to fully rely on my own mutex. It's just a bit cleaner imho. – Porush

Here is a paper, that implicitly lists support for MPI_THREAD_MULTIPLE for various other implementations: MVAPICH 1.9a2, IBM MPI 1.2, and Cray MPI. They also note the same problems with regard to OpenMPI and InfiniBand. They report that IntelMPI 4.0.3 does support Infiniband and MPI_THREAD_MULTIPLE in some situations but suffers from deadlocks. See section 5.1 starting at eq. (2) and keep in mind the requirement from section 3 that the underlying MPI library supports MPI_THREAD_MULTIPLE. – Dinky

But even in the "best" supported case, does it mean that one can use the same communicator simultaneously from to threads? – Optimum

@Optimum as long as the calls do not interfere with one another, multiple threads can use the same communicator simultaneously. – Mugger

By interfere you mean that they are writing to the same data, or that (the circumstantial tags) are erroneus and things like that? And just to confirm: this is true for MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE only, right? Also, this seems to be contradicting with this github.com/open-mpi/ompi/issues/2427#issuecomment-260954814, at least for collective communications. – Optimum

@alfC, yes, for example two MPI_Recv calls posted with the same source, tag, and communicator will interfere. Collective communications are an instance of this problem because many implementations, including Open MPI, use the same internal tag for all collective communications of a given type. That's also why the MPI standard says that each participating rank must issue collective operations in the exact same order, which might become problematic when more than one thread is involved. – Mugger

It looks like despite all intentions the end result is that mpi communication is not thread safe unless one can guarantee unique tags, and definitely not for collective communication. – Optimum

Answer 2

There are four levels of MPI thread safety, not all of them supported by every implementation: MPI_THREAD_SINGLE, MPI_THREAD_FUNNELED, MPI_THREAD_SERIALIZED and MPI_THREAD_MULTIPLE. The last one, which allows for a process to have multiple threads which may simultaneously call MPI functions, is probably the one you are interested in. So, first of all, you need to make sure your implementation supports MPI_THREAD_SERIALIZED.

The required level of thread safety must be specified by a call to MPI_Init_thread. After you have called MPI_Init_thread you should be able to safely call MPI functions in boost (POSIX) threads created on your own.

Marital

"The call to MPI_Init_thread initializes threading environment and actually creates the threads." -- the second part of that statement is false. – Mugger