mpiexec and python mpi4py gives rank 0 and size 1

Question

I have a problem with running a python Hello World mpi4py code on a virtual machine.

The hello.py code is:

#!/usr/bin/python
#hello.py
from mpi4py import MPI

comm = MPI.COMM_WORLD

size = comm.Get_size()
rank = comm.Get_rank()

print "hello world from process ", rank,"of", size

I've tried to run it using mpiexec and mpirun, but it is not running well. The output:

$ mpirun -c 4 python hello.py 
hello world from process  0 of 1
hello world from process  0 of 1
hello world from process  0 of 1
hello world from process  0 of 1

And from mpiexec:

$ mpiexec -n 4 python hello.py 
hello world from process  0 of 1
hello world from process  0 of 1
hello world from process  0 of 1
hello world from process  0 of 1

They seem not getting rank and size of comm. What can cause this? How to solve it?

mpiexec --version
mpiexec (OpenRTE) 1.6.5

mpirun --version
mpirun (Open MPI) 1.6.5

The system is Ubuntu 14.04 on the Virtal Machine.

Any ideas why? Thanks!

Provoke

possible duplicate of MPI_Rank return same process number for all process – Permutation

Most likely, your mpi4py is built against a different MPI implementation than the mpiexec/mpirun you're using to run the program. Does mpi4py.get_config() return compilers that are in the same path as your mpirun? – Permutation

mpi4py.get_config() returns {'mpicxx': '/usr/bin/mpicxx', 'mpif77': '/usr/bin/mpif77', 'mpicc': '/usr/bin/mpicc', 'mpif90': '/usr/bin/mpif90'} But I tried the same on my localhost and get the same output (the same config), and at my localhost the code works. – Provoke

great ; now where is mpirun/mpi exec? Eg, what's the output of which mpirun? – Permutation

$ which mpirun mpiexec /usr/bin/mpirun /usr/bin/mpiexec – Provoke

And whereis: mpirun: /usr/bin/mpirun.mpich /usr/bin/mpirun /usr/bin/mpirun.openmpi /usr/bin/X11/mpirun.mpich /usr/bin/X11/mpirun /usr/bin/X11/mpirun.openmpi /usr/share/man/man1/mpirun.1.gz mpiexec: /usr/bin/mpiexec.mpich /usr/bin/mpiexec.hydra /usr/bin/mpiexec /usr/bin/mpiexec.openmpi /usr/bin/X11/mpiexec.mpich /usr/bin/X11/mpiexec.hydra /usr/bin/X11/mpiexec /usr/bin/X11/mpiexec.openmpi /usr/share/man/man1/mpiexec.1.gz Sorry for formatting, I'm just getting know how to use Stack Oerflow – Provoke

Ok - it looks like there's both MPICH and OpenMPI versions (or at least stubs of versions) installed for MPI. This is the canonical source of this problem - you have an mpi4py compiled with one implementation and are trying to run it with another. Try either explicitly running with both mpirun.mpich and mpirun.openmpi, or completely uninstall one or the other (including all libraries) – Permutation

Yeah, the openmpi mpiexec version returned 0 and 1. The mpich mpiexec works great. I am really thankful for your help! – Provoke

Answer 1

As suggested above and in this question for C, this has to do with having mpirun coming from a different MPI than mpi4py was linked against.

In my case, and I suspect the same is true for many other Python users, this came from having originally installed mpi4py through conda, which pulled a non-system version of MPI into my conda -- i.e. which mpirun gave a path in my conda environment.

To solve the problem, I ran conda remove mpi4py and then pip install mpi4py, which seemed to rebuild mpi4py against the MPI in the conda environment and solved the issue.

Drawer

Worked for me. Had to module load OpenMPI before pip installation. – Lodestar

Answer 2

I had the same issue when running the python module emcee. It would give me an error:

"ValueError: Tried to create an MPI pool, but there was only one MPI process available. 
Need at least two."

The solution I found for my particular cluster was to use a different MPI. My code worked with intel-mpi and mpich2 but not openmpi. For this system, all I had to do was switch the MPI. In my PBS script I used module load mpich2 instead of module load openmpi. In this case mpiexec and mpirun worked correctly.

Flowerless

Switch what? Switch where? I am trying to understand your post, I have the same problem as OP. – Iso

I edited the answer to clarify that I switched the MPI for my particular system. – Flowerless