MPI_Cart_Shift.Corner Neighborhood

Question

I need to create a communicator with cube topology and then select face of the cube, using the MPI_Cart_Shift implemented messaging between processes that are on the brink. For example I am process with rank 0(R0) my neighborhoods are R2, R4, R6(lower face of cube). I can find R2 and R4, but I can't understand how to find R6. My code:

#include<mpi.h>
#include<stdio.h>

int main(int argc, char *argv[])
{
int rank, k;
int size; 
int ndims = 3; 
int source, dest;
int up,down,right,left,up3, down3;

int edges[6][4] = {{0,1,5,4},
                {4,5,7,6},
                {2,3,1,0},
                {6,7,3,2},
                {1,3,7,5},
                {0,2,6,7}};

int t, incep=0;
char processor_name[MPI_MAX_PROCESSOR_NAME];

MPI_Comm comm, comm3d;
int dims[3]={0,0,0}, coords[3]={0,0,0},
    periods[3]={1,1,1}, reorder = 0;


MPI_Status status;


int user_edge;

MPI_Init(&argc, &argv);

MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

MPI_Dims_create(size, ndims, dims);


  MPI_Cart_create(MPI_COMM_WORLD, ndims, dims, periods, reorder, &comm);


  MPI_Cart_coords(comm, rank, 3, coords);

 fflush(stdout);
     printf("Rank %d coordinates are %d %d %d\n", rank, coords[0], coords[1], coords[2]);
 MPI_Barrier(comm);

    int leftrank, rightrank;
    int downrank, uprank;

MPI_Comm_rank(comm, &rank);
    MPI_Cart_coords(comm, rank, 2, coords);
MPI_Cart_shift(comm, 0, -1, &downrank, &uprank); 
MPI_Sendrecv(buffer, 10, MPI_INT, downrank, 123, buffer2, 10, MPI_INT, uprank, 123, comm, &status);
MPI_Cart_shift(comm, 1, -1, &rightrank, &leftrank); 
    MPI_Sendrecv(buffer, 10, MPI_INT, leftrank, 123, buffer2, 10, MPI_INT, rightrank, 123, comm, &status);


printf("P:%d My neighbors are rightRank: %d downRank:%d leftRank:%d upRank:%d diagonal:%d diagonalX:%d\n", rank,rightrank,downrank,leftrank,uprank,diagonal,diagonalX);



MPI_Finalize();

return 0;

}

I will try add something like this MPI_Cart_shift(comm, 2, 1, &diagonal, &diagonalX); But for R0 it show me R1 and it i understand.... How can I get corner neighborhoods?

Dituri

Answer 1

You can use MPI_Cart_rank to find the information you need.

int MPI_Cart_rank(MPI_Comm comm, const int coords[], int *rank)

Here comm is a communicator with Cartesian topology. coords is a integer array (the size of this array being the number of dimensions of the Cartesian topology) containing the coordinate of a process (in your case, 1,1,0 for R6). Then the output rank would return the global rank of that process you can use in subsequent communication.

BTW, MPI_Cart_coords performs the opposite direction, i.e. from rank to coordinate.

Montanez

Answer 2

MPI_Cart_shift can only find neighbours in any of the main directions, but not diagonally - you have to implement it on your own, which is not that hard to do. The Cartesian shift itself is nothing more than a convenience function that wraps calls to MPI_Cart_coords and MPI_Cart_rank:

//
// The following is equivalent to
// MPI_Cart_shift(cart_comm, i, disp, &rank_source, &rank_dest);
//

// Obtain the rank of the calling process and translate it into coordinates
int rank, coords[ndims];
MPI_Comm_rank(cart_comm, &rank);
MPI_Cart_coords(cart_comm, rank, ndims, coords);

int mycoord_i = coords[i];

// Compute the coordinates of the destination in direction i and convert them into rank
coords[i] = mycoord_i + disp;
// Take care of non-periodic dimensions
if (!periods[i] && (coords[i] >= dims[i] || coords[i] < 0))
   rank_dest = MPI_PROC_NULL;
else
   MPI_Cart_rank(cart_comm, coords, &rank_dest);

// Compute the coordinates of the source in direction i and convert them into rank
coords[i] = mycoord_i - disp;
// Take care of non-periodic dimensions
if (!periods[i] && (coords[i] >= dims[i] || coords[i] < 0))
   rank_source = MPI_PROC_NULL;
else
   MPI_Cart_rank(cart_comm, coords, &rank_source);

Note how going out-of-bound should be explicitly handled for non-periodic dimensions. For periodic ones, MPI_Cart_rank itself does the folding of the coordinates. Your Cartesian topology is periodic, therefore the checks could be skipped (not recommended).

You can easily adapt the code above to produce a shift in two or more dimensions, then use it to enumerate over all possible combinations of displacements and directions to find the ranks that lie on a given face. For example:

int rank, coords[ndims];
MPI_Comm_rank(cart_comm, &rank);
MPI_Cart_coords(cart_comm, rank, ndims, coords);
int my_coord1 = coords[1];
int my_coord2 = coords[2];
coords[1] = my_coord1 + 1;
coords[2] = my_coord2 + 1;
MPI_Cart_rank(cart_comm, coords, &rank_dest);
coords[1] = my_coord1 - 1;
coords[2] = my_coord2 - 1;
MPI_Cart_rank(cart_comm, coords, &rank_source);

will give you the rank of R6 in rank_dest when executed by R0. If you extract the code in a separate routine and call it MyMPI_Cart_shift2:

MyMPI_Cart_shift2(cart_comm, 1, 1, 2, 1, &rank_source, &rank_dest);

Another option would be to use MPI_Cart_sub to split the Cartesian topology into slabs, parallel to the face. You can then directly perform communication involving only ranks in any given slab.

Nolte

Answer 3

You forgot to label R5. There is not direct approach that I can think of. But here is an indirect way:

(1) For any rank whose coordinates are (A,y,z) consider coordinates : (A,y+1,z+1), (A,y-1,z-1), (A,y+1,z-1) and (A,y-1,z+1) - these can be the 4 diagonal neighbours. Clearly, when considering a minus for e.g. z-1, it is clear that z-1 >= 0 and when considering a plus for e.g. y+1 <= (dimension_in_y - 1). For example consider R5 (which you forgot to label). R5 according to MPI has coordinates R5(0,1,0). It can have 4 diagonal neighbours : (0,1+1,0+1), (0,1-1,0-1),(0,1+1,0-1),(0,1-1,0+1) - out of which only the fourth one exists (which is clear as no neighbour can have negative coordinates and each coordinate must be 1 less than dimension in that direction). Next simply query MPI_coordinate_to_rank().

(2) Another method is to query the right and left neighbours about their front and back neighbours (this will be bad performance-wise).

(3) To generalise it to 8 neighbours - you'll need to vary the X coordinate as well - but your question is limited to the face of a cube.

Ftc

Please check the process coordinates. They are NOT in accordance with the MPI cartesian coordinates. – Ftc