struct serialization in C and transfer over MPI

Question

I have defined a custom struct which I need to send over to another MPI process using the MPI_Bsend (or MPI_Send).

Here is the struct:

struct car{
  int shifts;
  int topSpeed;
}myCar;

The issue is that apart from primitive types MPI doesn't seem to support direct "transmission" of complex data types like the struct shown above. I've heard that I might have to use "serialization".

How should I approach this and successfully send over myCar to process 5?

Jaquez

Answer 1

Jeremiah is right - MPI_Type_create_struct is the way to go here.

It's important to remember that MPI is a library, not built into the language; so it can't "see" what a structure looks like to serialize it by itself. So to send complex data types, you have to explicitly define its layout. In a language that does have native support for serialization, a set of MPI wrappers can concievably make use of that; mpi4py for instance makes use of python's pickle to transparently send complex data types; but in C, you have to roll up your sleeves and do it yourself.

For your structure, it looks like this:

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <stddef.h>

typedef struct car_s {
        int shifts;
        int topSpeed;
} car;

int main(int argc, char **argv) {

    const int tag = 13;
    int size, rank;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if (size < 2) {
        fprintf(stderr,"Requires at least two processes.\n");
        exit(-1);
    }

    /* create a type for struct car */
    const int nitems=2;
    int          blocklengths[2] = {1,1};
    MPI_Datatype types[2] = {MPI_INT, MPI_INT};
    MPI_Datatype mpi_car_type;
    MPI_Aint     offsets[2];

    offsets[0] = offsetof(car, shifts);
    offsets[1] = offsetof(car, topSpeed);

    MPI_Type_create_struct(nitems, blocklengths, offsets, types, &mpi_car_type);
    MPI_Type_commit(&mpi_car_type);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (rank == 0) {
        car send;
        send.shifts = 4;
        send.topSpeed = 100;

        const int dest = 1;
        MPI_Send(&send,   1, mpi_car_type, dest, tag, MPI_COMM_WORLD);

        printf("Rank %d: sent structure car\n", rank);
    }
    if (rank == 1) {
        MPI_Status status;
        const int src=0;

        car recv;

        MPI_Recv(&recv,   1, mpi_car_type, src, tag, MPI_COMM_WORLD, &status);
        printf("Rank %d: Received: shifts = %d topSpeed = %d\n", rank,
                 recv.shifts, recv.topSpeed);
    }

    MPI_Type_free(&mpi_car_type);
    MPI_Finalize();

    return 0;
}

Locality

Thank you for your very comprehensive and quick response. I really appreciate it. You've got me completely covered. (However I think you forgot to include <stddef.h> on the top cause otherwise the compiler gives errors..) – Jaquez

You're right - needed for the offsetof(). I've updated the code appropriately. – Locality

Thanks for this helpful answer. I have a question: How should your example be changed if one of the members of the struct, say shifts, is an array? (int shifts[2];) – Thymol

Answer 2

Although Jonathan Dursi's answer is correct, it is overly complicated. MPI provides simpler and less general type constructors more suitable for your problem. MPI_Type_create_struct is ONLY needed when you have different base types (e.g., an int and a float).

For your example, several better solutions exist:

• Assuming that the two integers are aligned in a contiguous memory area (i.e., like an array of integers), you don't need a derived datatype at all. Just send/receive two elements of type MPI_INT with the address of a variable of type car to be used as the send/receive buffer:

  MPI_Send(&send, 2, MPI_INT, dest, tag, MPI_COMM_WORLD);
  MPI_Recv(&recv, 2, MPI_INT, src, tag, MPI_COMM_WORLD, &status);
  

• If you want to use a derived datatype (e.g., for readability or the fun of it), you can use MPI_Type_contiguous which corresponds to arrays:

  MPI_Type_contiguous(2, MPI_INT, &mpi_car_type);
  

• In case the two integers are aligned differently (most likely not the case, but it is machine dependent and MPI implementations exist for a lot of different platforms), you can use MPI_Type_indexed_block: It takes an array of displacements (like MPI_Type_create_struct), but only one oldtype argument and the block-length of every block is 1 by definition:

  MPI_Aint offsets[2];
  offsets[0] = offsetof(car, shifts) ; //most likely going to be 0 
  offsets[1] = offsetof(car, topSpeed);
  MPI_Type_indexed_block(2, offsets, MPI_INT);
  

While the other solution is semantically correct, it is a lot harder to read and may incur a large performance penalty.

Arkwright

Answer 3

Look at MPI_Type_create_struct to build a custom MPI datatype for your object. An example of using it is at http://beige.ucs.indiana.edu/I590/node100.html.

Beaconsfield

I am still a bit confused..!Suppose I define the MPI structure and now want to use it. The link you gave states: MPI_Type_create_struct(5, array_of_block_lengths, array_of_displacements, array_of_types, &new_type); Should I now do something like myCar=&new_type? – Jaquez

And more to the point...Could you please give me a simple but concrete example of creating and transmitting the specific struct? – Jaquez

Problem solved. The link you provided gives all the "theory" but may easily confuse amateur programmers due to the displacements and low level details. However it seems to precisely describe the mechanism underneath. – Jaquez

The link is broken. Can you provide an example in your answer? – Incisor

Answer 4

int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)

OpenMPI will send count * sizeof(datatype) contiguous bytes starting at buf to allow sending things like int arrays. For example, if you declare a 10 int array int arr[10], you can send with

MPI_Send(arr, 10, MPI_INT, 1, 0, MPI_COMM_WORLD);

and receive similarly. Since buf is a void pointer we can abuse this to send structs by sending sizeof(my_struct) bytes and casting back as struct on the receiving end. Here is an example:

#include "mpi.h"
#include <stdio.h>

typedef struct 
{
    char a;
    int b;
    short c;
} my_struct;


int main (int argc, char *argv[])
{
    int  numtasks, taskid;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &taskid);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);


    if (taskid == 0) 
    {
        my_struct m;
        m.a = '!';
        m.b = 1234;
        m.c = 5678;

        MPI_Send(&m, sizeof(my_struct), MPI_CHAR, 1, 0, MPI_COMM_WORLD);
    }
    else 
    {
        my_struct m;
        MPI_Recv(&m, sizeof(my_struct), MPI_CHAR, 0, 0, MPI_COMM_WORLD, 
                 MPI_STATUS_IGNORE);
        printf("%c %d %d\n", m.a, m.b, m.c); 
    }

    MPI_Finalize();
}

Since C arrays store data contiguously, we can even send arrays of structs similarly to how we malloc an array of structs. So if you had a my_struct m_array[10] you would send (and receive similarly) with

MPI_Send(m_array, sizeof(my_struct) * 10, MPI_CHAR, 1, 0, MPI_COMM_WORLD);

Bisayas

What are the downsides of this solution? Does MPI treat the data differently if it knows its structure and if yes then how? – Rebbeccarebe

I don't know how MPI sends MPI-created struct types but a reasonable implementation is to send the whole struct as I am doing with the custom type just used for calculating offsets. Sending smaller pieces increases granularity which may or may not speed things up depending on the overhead of each message and things like buffer sizes. – Bisayas