Mvapich2 buffer aliasing

Question

I am launched an MPI program with MVAPICH2 and got this error:

Fatal error in PMPI_Gather:
Invalid buffer pointer, error stack:
PMPI_Gather(923): MPI_Gather() failed
PMPI_Gather(857): Buffers must not be aliased

There are two ways I think I could solve this:

1. Rewrite my MPI program (use different buffers)
2. Disable checking buffer aliasing

Do someone know how I could do this with MVAPICH2? Some compiler option, parameter, environmental variable, etc?

Something like MV2_NO_BUFFER_ALIAS_CHECK, but it does not work.

Pituri

FYI, when you're writing your next question or answer, take a look at how to format it properly. Generally on StackOverflow, you shouldn't use HTML formatting to enforce line breaks. You can look at how I've edited your question to see how to do the formatting correctly. There's also buttons at the top of the text box that help you format more easily. – Beitnes

Answer 1

What you're doing is an incorrect program and you should rewrite your code to use separate buffers

Alternatively, you might be able to use MPI_IN_PLACE if you want to use the same buffer as both the input and output values of your MPI_GATHER. Without seeing your code, I can't tell you how you could do that. You can check out some documentation about MPI_GATHER and read more about how MPI_IN_PLACE works and see if that solves your problem.

Beitnes

It's shocking that even in an "industrial-level" benchmarking suite such as SPEC MPI2007 there are several benchmarks that fail to run on modern systems due to operations such as MPI_Allgather(&array[rank], 1, MPI_INT, array, 1, MPI_INT, ...);. I guess, the "works for me" syndrome is more widespread that it might look like. – Electrodeposit

That's a loophole that was just recently closed in MPICH. There's an environment variable you can set to not check it though. – Beitnes

In our case the problem runs deeper. The MPI library does not explicitly check for overlapping buffers (only tests for pointer equality) but then the memcpy() it uses to copy the local chunk into the local gather buffer call barfs. Rewriting the code to be compliant with the MPI standard is the only way to go. – Electrodeposit

That's what we did until recently. We have more comprehensive checks now but it's probably gonna catch quite a few incorrect uses. – Beitnes

Nice to see someone from ANL around here :) – Electrodeposit

We're lurking everywhere. :) – Beitnes

Thank you for reply. Unfortunately I can't correct code because it is very big file written in FORTRAN. I know about MPI_IN_PLACE. I was trying to correct calls MPI_GATHER by this way. But it is hard for me, it is not my code. And as a result of my fixes program crashed. I looking for some option on Mvapich which can help me. But seems it is wrong way. – Pituri

There was an option added to MPICH (which is the basis for MVAPICH) called MPICH_COLL_ALIAS_CHECK that you can set to 0. I doubt that's made it into MVAPICH yet though since those were just added in 3.1 I believe and I don't think that MVAPICH has pulled that down yet. – Beitnes

@WesleyBland Unfortunately it doesn't help. Anyway thanks a lot for reply! – Pituri