Generic MPI code

Question

I want to create a generic MPI method, let's say a bcast for a specific object. but I need to convert primitive types to MPI_Data types ? any idea how I can do it ?

template <typename T>
void bcast_data(std::vector<T> vec) 
{
...
}

I need to use MPI_INT for int, MPI_DOUBLE for double , ... so I need a type conversion method, I thought of creating an enum of dataypes that could give me the MPI_datatypes, but it requires passing the type as an input argument.

any idea ?

Thanks

Leadsman

Have a look at boost::mpi. It's basically a c++ wrapper for MPI and does exactly that. – Allelomorph

Answer 1

You can use "type traits" idiom to serialize a generic object T. This gives you the advantage to be able to add support for new types without changing the implementation.

Take a look at this MPI wrapper I wrote years ago: https://github.com/motonacciu/mpp.

You want to define a type trait like the following:

template <class T>
struct mpi_type_traits {
    typedef T element_type;
    typedef T* element_addr_type;

    static inline MPI_Datatype get_type(T&& raw);
    static inline size_t get_size(T& raw);
    static inline element_addr_type get_addr(T& raw);
};

and provide specialization for concrete types, e.g. an std::vector<T> as follows:

template <class T>
struct mpi_type_traits<std::vector<T>> {

    typedef T element_type;
    typedef T* element_addr_type;

    static inline size_t get_size(std::vector<T>& vec) {
       return vec.size();
    }

    static inline MPI_Datatype get_type(std::vector<T>&& vec) {
        return mpi_type_traits<T>::get_type( T{} );
    }

    static inline element_addr_type get_addr(std::vector<T>& vec) {
        return mpi_type_traits<T>::get_addr( vec.front() );
    }
};

The last thing you need to do is to implement your MPI method and use the type traits, e.g. when calling an MPI_Send:

template <class T>
void send(T &&value, ...) {
   MPI_Send(mpi_type_traits<T>::get_addr(value),
            mpi_type_traits<T>::get_size(value),
            mpi_type_traits<T>::get_type(value), ...);
}

Scumble

Great answer. One thing that seems to invite trouble is using the actual object to get type information (e.g. vec.size()). This means that if the sender and receiver have different objects, they must ensure to produce compatible types. E.g.. the receiver has to know the sender size and resize accordingly before receiving. I wonder how do you handle this in your implementation? – Maunsell

This is not much different from the typical use case of MPI. Send and receiver need to know before hand the lenght and type of objects being sent. You can solve this by designing a protocol where a descriptor (containing size and maybe type information) for the object is sent before the actual object. – Scumble

I don't generally disagree, but 1) std::vector is usually used without explicitly setting / knowing the size (through begin,end,push_back). Taking the size is also invisible to the user in your solution. 2) In Boost.MPI you can actually pass a reference to an empty vector to receive in it - at significant performance cost. 3) You could use MPI_Probe to receive a vector without knowing it's size and without having an additional message / serialization. I do wonder if there are any C++ abstractions that do support the latter. – Maunsell

Answer 2

I think the Boost feature get_mpi_datatype should offer this functionality. I recommend building on such a sophisticated library rather than home brewed code whenever you can.

If you are looking for a light-weight generic solution that does not rely on Boost, one might extend apramc's idea to all current MPI data types with a constexpr function with type_traits such that the corresponding MPI data type is already evaluated at compile time as follows (click here for the Gist, C++17 required)

#include <cassert>
#include <complex>
#include <cstdint>
#include <type_traits>

#include <mpi.h>


template <typename T>
[[nodiscard]] constexpr MPI_Datatype mpi_get_type() noexcept
{
    MPI_Datatype mpi_type = MPI_DATATYPE_NULL;
    
    if constexpr (std::is_same<T, char>::value)
    {
        mpi_type = MPI_CHAR;
    }
    else if constexpr (std::is_same<T, signed char>::value)
    {
        mpi_type = MPI_SIGNED_CHAR;
    }
    else if constexpr (std::is_same<T, unsigned char>::value)
    {
        mpi_type = MPI_UNSIGNED_CHAR;
    }
    else if constexpr (std::is_same<T, wchar_t>::value)
    {
        mpi_type = MPI_WCHAR;
    }
    else if constexpr (std::is_same<T, signed short>::value)
    {
        mpi_type = MPI_SHORT;
    }
    else if constexpr (std::is_same<T, unsigned short>::value)
    {
        mpi_type = MPI_UNSIGNED_SHORT;
    }
    else if constexpr (std::is_same<T, signed int>::value)
    {
        mpi_type = MPI_INT;
    }
    else if constexpr (std::is_same<T, unsigned int>::value)
    {
        mpi_type = MPI_UNSIGNED;
    }
    else if constexpr (std::is_same<T, signed long int>::value)
    {
        mpi_type = MPI_LONG;
    }
    else if constexpr (std::is_same<T, unsigned long int>::value)
    {
        mpi_type = MPI_UNSIGNED_LONG;
    }
    else if constexpr (std::is_same<T, signed long long int>::value)
    {
        mpi_type = MPI_LONG_LONG;
    }
    else if constexpr (std::is_same<T, unsigned long long int>::value)
    {
        mpi_type = MPI_UNSIGNED_LONG_LONG;
    }
    else if constexpr (std::is_same<T, float>::value)
    {
        mpi_type = MPI_FLOAT;
    }
    else if constexpr (std::is_same<T, double>::value)
    {
        mpi_type = MPI_DOUBLE;
    }
    else if constexpr (std::is_same<T, long double>::value)
    {
        mpi_type = MPI_LONG_DOUBLE;
    }
    else if constexpr (std::is_same<T, std::int8_t>::value)
    {
        mpi_type = MPI_INT8_T;
    }
    else if constexpr (std::is_same<T, std::int16_t>::value)
    {
        mpi_type = MPI_INT16_T;
    }
    else if constexpr (std::is_same<T, std::int32_t>::value)
    {
        mpi_type = MPI_INT32_T;
    }
    else if constexpr (std::is_same<T, std::int64_t>::value)
    {
        mpi_type = MPI_INT64_T;
    }
    else if constexpr (std::is_same<T, std::uint8_t>::value)
    {
        mpi_type = MPI_UINT8_T;
    }
    else if constexpr (std::is_same<T, std::uint16_t>::value)
    {
        mpi_type = MPI_UINT16_T;
    }
    else if constexpr (std::is_same<T, std::uint32_t>::value)
    {
        mpi_type = MPI_UINT32_T;
    }
    else if constexpr (std::is_same<T, std::uint64_t>::value)
    {
        mpi_type = MPI_UINT64_T;
    }
    else if constexpr (std::is_same<T, bool>::value)
    {
        mpi_type = MPI_C_BOOL;
    }
    else if constexpr (std::is_same<T, std::complex<float>>::value)
    {
        mpi_type = MPI_C_COMPLEX;
    }
    else if constexpr (std::is_same<T, std::complex<double>>::value)
    {
        mpi_type = MPI_C_DOUBLE_COMPLEX;
    }
    else if constexpr (std::is_same<T, std::complex<long double>>::value)
    {
        mpi_type = MPI_C_LONG_DOUBLE_COMPLEX;
    }
    
    assert(mpi_type != MPI_DATATYPE_NULL);
    return mpi_type;    
}

You can call the MPI commands then like e.g.

template <typename T>
void mpi_exchange_inplace(std::vector<T>& vec, int const length, MPI_Comm const& icomm)
{   
    MPI_Alltoall(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, vec.data(), length, mpi_get_type<T>(), icomm); 
    return;
}

Zolazoldi

Answer 3

I used something like this, it is definitely not a complete answer as it leaves some types uncovered. but it works for my case

template<typename T>
MPI_Datatype get_type()
{
    char name = typeid(T).name()[0];
    switch (name) {
        case 'i':
            return MPI_INT;
        case 'f':
            return MPI_FLOAT;
        case 'j':
            return MPI_UNSIGNED;
        case 'd':
            return MPI_DOUBLE;
        case 'c':
            return MPI_CHAR;
        case 's':
            return MPI_SHORT;
        case 'l':
            return MPI_LONG;
        case 'm':
            return MPI_UNSIGNED_LONG;
        case 'b':
            return MPI_BYTE;
    }
}

Leadsman

Answer 4

I once came up with a solution that was very similar to the ones already shown, but with a slight advantage. The idea is to use a templated function and add specializations to resolve the type to the corresponding MPI type:

namespace mpiUtil { // Namespace for convenience

    template <typename T>
    MPI_Datatype resolveType();

    template <>
    MPI_Datatype resolveType<double>()
    {
        return MPI_DOUBLE;
    }

    // ... add a specialization for all other types

    template <typename T>
    int autoSend(const T *items, int count, const int dest, const int tag, 
                 const MPI_Comm comm)
    {
        return MPI_Send(items, count, resolveType<T>(), 0, tag, comm);
    }
    
    // You can repeat this procedure for other MPI functions
}

The main advantage is that it is easily extensible for custom types. For example if you have a class and an array of said class you want to transfer, you can add a custom type to the MPI system and inject a specialization of resolveType() for this class into the mpiUtil namespace.

Linneman