Creating an MPI_Datatype for a structure containing pointers

Question

I have the following structure.

typedef struct
{
 int *Ai;
 double *Ax;
 int nz;
}column;

I want to transfer this structure using MPI_Send and MPI_Receive. How do I create an MPI_Datatype for this structure?

Lantana

Why would you ever want to send pointers between MPI processes? They are non-portable in distributed memory systems. – Necrolatry

Answer 1

MPI is designed to work with arrays of structures rather that with structures of arrays.

The MPI_Hindexed that @suszterpatt proposed is a terrible hack. It will only allow you to send one element of the structure type and only the element that was used to define the MPI data type. For other variables of the same structure type it is mostly guaranteed that the computed offsets will be wrong. Besides Hindexed types use one and the same MPI data type for all elements and thus does not allow you to send both ints and doubles.

The wise thing to do is to transform your program to use arrays of structures:

 typedef struct
 {
    int i;
    double z;
 } point;

 typedef struct
 {
    point *A;
    int nz;
 } column;

Now you can create an MPI structured type point_type and use it to send nz elements of that type giving column.A as the buffer address:

 int lens[3];
 MPI_Aint base, disps[2];
 MPI_Datatype oldtypes[2], point_struct, point_type;

 MPI_Get_address(&point, disps);
 MPI_Get_address(&point.z, disps+1);
 base = disps[0];

 lens[0] = 1; disps[0] = MPI_Aint_diff(disps[0], base); oldtypes[0] = MPI_INT;
 lens[1] = 1; disps[1] = MPI_Aint_diff(disps[1], base); oldtypes[1] = MPI_DOUBLE;
 MPI_Type_create_struct(2, lens, disps, oldtypes, &point_struct);
 MPI_Type_create_resized(point_struct, 0, sizeof(point), &point_type);
 MPI_Type_commit(&point_type);

 MPI_Send(column.A, column.nz, point_type, ...);

This first creates an MPI datatype point_struct that describes the layout of the structure members, but does not account for any padding at the end and therefore cannot be used to reliably send an array of such structures. Therefore, a second datatype point_type with the correct extent is created using MPI_Type_create_resized.

On the receiver side you would peek the message with MPI_Probe, extract the number of elements with MPI_Get_count with a type of point_type (that goes straight to the nz field), allocate the A field and use it in MPI_Recv to receive the nz elements:

 MPI_Status status;
 MPI_Probe(source, tag, comm, &status);
 MPI_Get_count(&status, point_type, &column.nz);
 if (nz == MPI_UNDEFINED)
   ... non-integral message was received, do something
 column.A = (point *)malloc(column.nz*sizeof(point));
 MPI_Recv(column.A, column.nz, point_type, source, tag, comm, MPI_STATUS_IGNORE);

If that code change is impossible you can still go through the intermediate step of transforming your structure before sending it, a process usually called (un-)marshaling. In your case do something like this (I assume that you store the number of array elements in both Ai and Ax in the nz field):

 point *temp = (point *)malloc(nz*sizeof(point));
 for (int i = 0; i < column.nz; i++)
 {
   temp[i].i = column.Ai[i];
   temp[i].z = column.Az[i];
 }
 MPI_Send(temp, nz, point_type, ...);
 free(temp);

On the receiver side you must do the opposite: allocate a large enough buffer that can hold the structure, receive the message in it and then do the opposite transformation.

Once again, you do not need to transmit the actual value of nz since it can be easily extracted from the length of the message using MPI_Get_count.

Anaxagoras

Thanks, this was helpful. I used an intermediate representation like you suggested and it was much better. – Lantana

Are you sure about the "disp[0]"? If the zero element is a character I wouldn't know whether it was in the left or right byte of the smallest alignment unit. In other words: I would compute its offset explicitly. – Renz

@VictorEijkhout, the C standard (specifically ISO/IEC 9899:1999, haven't read the older documents) mandates that a pointer to a structure is a pointer to its first member, therefore the offset of the first member if computed likewise will be zero; padding at the beginning of a structure is not allowed (§6.7.2.1(13)). But the code is terrible anyway - pointer arithmetic with different pointer types and use of MPI_UB... – Anaxagoras

Answer 2

Sending pointers to another machine is pointless (no pun intended). Due to virtual addressing, the pointer will likely point to an invalid memory location on the receiving machine, and even if not, you haven't actually sent the data that it was pointing to.

However, with proper use of MPI_Address() and an MPI_Hindexed datatype, it is possible to describe the memory layout of your data (I'm assuming that your pointers point to dynamic arrays). E.g. if Ai points to 3 ints, and Ax points to 5 doubles, you'll need a Hindexed type with 3 blocks: 3 MPI_INTs, 5 MPI_DOUBLEs, and 1 MPI_INT, with the offsets acquired using MPI_Address().

Don't forget to redefine and recommit the datatype if you change the number of items to be sent or reallocate the arrays entirely. And if you're sending multiple structs, you'll have to define and commit this datatype for each one, since your MPI datatype is specific to one particular instance of these structs.

Also keep in mind that you'll have to do some similarly tricky unpacking on the receiving end if you want to recreate the original struct.

Lesser

Thanks for the help. I tried this but it was a round about way and was creating too many datatypes. So I changed my data representation instead. Yes the pointers were to dynamic arrays. – Lantana

Answer 3

"The wise thing to do is to transform your program to use arrays of structures"

Often that's conceptually also better.

I would like to point out another mechanism: using MPI_Pack and MPI_Unpack. For instance, with the original structure you could pack the first integer, then pack the two arrays. The receiver would unpack the integer and then know how many of the other thingies to unpack.

This is also a good solution if your object is not directly accessible but can only be accessed through an iterator or so.

Renz