Fine grain control over rows distribution among locales in multi-locale program in Chapel

Question

I am trying to implement an SOR, successive over relaxation, program in Chapel for multi-locale, but with local memory so I want to distribute rows among the locales explicitly. I already have reshaped targetlocales to 1D, but now I am not sure how I can have control over the rows distribution among the locales.

I am very much used to MPI so I will give an example of what I want to achieve in accordance to MPI. Is there a way I can specify that I want to distribute all the array rows among locales as in #rows / #locales and the remaining rows on the last locales?

Issue I face in chapel at moment is:

• locales = 2 pattern was 5 rows on locale - 1, 5 rows on locale - 2
• locales = 3 pattern was 4 rows on locale - 1, 3 rows on locale - 2, 3 rows on locale - 3
• locales = 4 pattern was 3 rows on locale - 1, 2 rows on locale - 2, 3 rows on locale - 3, 2 rows on locale - 4
• locales = 8 pattern was 2 rows on locale - 1, 1 row on locale - 2, 1 row on locale - 3, 1 row on locale - 4, 2 rows on locale - 5, 1 row on locale - 6, 1 row on locale - 7, 1 row on locale - 8. The pattern changes as the #locales increases.

Since my array size will vary for each experiment, I want to have control over row distribution since its local memory multi-locale implementation. I would have to copy back and receive rows from neighboring locales.

Mooneye

Answer 1

As background, the Chapel language itself does not dictate how an array's rows are distributed between the locales; rather, it is the definition of a domain map—a user-defined type that maps a domain's indices and array's elements to locales—that does so. The current Block distribution that's part of Chapel's standard modules does not currently have a way of specifying how the blocking is done apart from the default which you describe above. However, with effort, one could write their own domain map—or customize the BlockDist module—to get a different distribution.

As I understand it, you want ceiling(rows / locales) on each of the initial locales and then the remaining elements on the final locale? This would be a reasonable thing to submit as a feature request on our GitHub issues page. Or if you wanted to tackle it yourself, you could ask for help getting started with the effort on Chapel's Gitter channel or mailing lists (see the user or developer resources pages for links to both).

Supercolumnar

As an editorial comment, I'll add that if your desire to control the per-locale block size is so that you can transliterate an MPI code into Chapel in a 1:1 fashion then it may be worth changing your thinking to make better use of Chapel's features (e.g., use StencilDist and have it communicate between neighbors rather than trying to do it manually yourself). OTOH, if your reason is that you need a specific layout in order to interoperate with an existing MPI library or code that requires a particular layout, that'd be a completely valid reason to extend Block in this way. – Supercolumnar