Distribute 2D array row wise among Locales in Chapel

Question

I am learning Chapel and have worked with blockdist but I can't figure out how can I distribute a 2-dimension array in row wise fashion among locales.

Titanesque

Answer 1

The key is to pass a reshaped Locales array as the targetLocales argument to Block. This is explained further below.

Here's a simple example of distributing a 2D array in a row-wise fashion:

use BlockDist;

// Using a 2D targetLocales rather than the default 1D Locales argument
var targetLocales = reshape(Locales, {0..#numLocales, 0..0});

const Space = {1..4, 1..4};
const D: domain(2) dmapped Block(boundingBox=Space, targetLocales=targetLocales) = Space;
var A: [D] int;

forall a in A do
  a = a.locale.id;

writeln(A);

Sample outputs:

 ./row-wise -nl 4

 0 0 0 0
 1 1 1 1
 2 2 2 2
 3 3 3 3

 ./row-wise -nl 2

0 0 0 0
0 0 0 0
1 1 1 1
1 1 1 1

By default, distributions will use the built-in Locales array as the targetLocales argument, which specifies how to paritition the elements of an array across the locales within a particular domain map, e.g. Block.

Since Locales is a 1D array, and you're distributing a 2D array, the Block distribution wraps Locales like so:

1D targetLocales:

0 1 2 3 -> 0 1
           2 3

So an array of shape (4,4) would get mapped to 4 locales as:

  0 0 1 1
  0 0 1 1
  2 2 3 3
  2 2 3 3

By providing a 2D targetLocales argument, we can tell Block explicitly how we'd like the elements to be mapped to locales, rather than rely on the wrapping. Passing a targetLocales array of locales with a shape of (4,1), will result in the desired row-wise distribution:

2D targetLocales:

   0
   1
   2
   3

So an array of shape (4,4) would get mapped to 4 locales as:

 0 0 0 0
 1 1 1 1
 2 2 2 2
 3 3 3 3

This concept applies to other distributions as well.

Abydos

Dear Ben, thank you for the explanation. I understand it very well now. I have one more question to ask regarding figuring out the range of indices for on locales. I tried below thing: I took a 10x10 array and experiment with distributing it over locals from range of 1 to 10. But unlike MPI, I can't find a fix pattern for range of indices which can be calculated on each locale. – Titanesque

For #locales = 2 pattern was 5 rows on locale - 1, 5 rows on locale - 2, #locales = 3 pattern was 4 rows on locale - 1, 3 rows on locale - 2, 3 rows on locale - 3, #locales = 4 pattern was 3 rows on locale - 1, 2 rows on locale - 2, 3 rows on locale - 3, 2 rows on locale - 4, #locales = 8 pattern was 2 rows on locale - 1, 1 row on locale - 2, 1 row on locale - 3, 1 row on locale - 4, 2 rows on locale - 5, 1 row on locale - 6, 1 row on locale - 7, 1 row on locale - 8. The pattern changes as the #locales increases. – Titanesque

I am very much used to of the MPI so I was wondering if there is a way I can decide on that I want to distribute all the array rows among locales as in #rows / #locales and the remaining rows on the last locales. Any pointers in this direction will be much appreciated. thanks – Titanesque

its there a build in function which can provide me the index of first and last row index mapped to a locale? I am trying to implement stencil program for multi locale using local memory. – Titanesque

Hi Robin - For fine grained control of the distribution of rows to locales, you can make your targetLocales be Nx1 for an NxN matrix, instead of numLocales x 1. This way, you can explicitly set the locale for each row. The RangeChunk library provides some library functions for chunking up the elements automatically with a few different remainder policies: chapel-lang.org/docs/modules/packages/RangeChunk.html – Abydos

I am happy to provide an example of this, but I think that would be more suitable in a separate stackoverflow question. Feel free to post a new question if you're interested in more details. – Abydos

re: stencil program -- I don't know that one off the top of my head. This may also be worth a separate question if you're not finding an answer. That said, you may be interested in utilizing the Stencil Distribution, which includes ghost / fluff cell support built in: chapel-lang.org/docs/latest/modules/dists/StencilDist.html – Abydos

Dear Ben, I have posted a new question : #53465214 – Titanesque