mpirun - not enough slots available

Question

Usually when I use mpirun, I can "overload" it, using more processors than there acctually are on my computer. For example, on my four-core mac, I can run mpirun -np 29 python -c "print 'hey'" no problem. I'm on another machine now, which is throwing the following error:

$ mpirun -np 25 python -c "print 'hey'"
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 25 slots 
that were requested by the application:
  python

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

Why isn't "overclocking" mpirun working here? Is there a way I can overcome this error message and successfully run with more processors than are available?

Numen

You probably mean oversubscribe, rather than overload or overclock it. BTW, which MPI implementation do you use? – Stoush

@Stoush I have gotten this question on a couple of mpi-related questions, and the answer is that I just don't know. Is it possible for a non-administrator to find this information? – Numen

mpirun --version yields (Open MPI) 1.7.3 – Numen

That's pretty weird. I'm using OpenMPI 1.10 on a quad-core system and your mpirun command works like a charm. – Stoush

Could you try with this information regarding oversubscribing on openmpi using a hostfile ? open-mpi.org/faq/?category=running#oversubscribing – Stoush

I saw that and was confused. What exactly would I try? – Numen

Answer 1

According to https://www.open-mpi.org/faq/?category=running#oversubscribing you could oversubscribe your node using a hostfile. Before proceeding, be careful that this way you can severely degrade the performance of the node. Also, if the system you use to run the application is using a queue system, this may not be valid.

First create a hostfile (named hostfile) containing

localhost slots=25

The simply run your application like

mpirun --hostfile hostfile -np 25 python -c "print 'hey'"

Stoush

The link you referenced, specifically says one shouldn't state more slots than available in the machine. Slots should state the actual quantity of processors available (or less) and, to oversubscribe, the --oversubscribe flag should be used – Anticoagulant

I meet with "Open RTE was unable to open the hostile xxx". But the file was there. Any help? – Stogy

Answer 2

Apparently oversubscribing can be attained using the "--oversubscribe" option with mpirun - did the trick for me with running torque/maui

Certainty

This seems a lot nicer than the accepted answer. Note that the --oversubscribe flag is a feature of OpenMPI 3.x. This flag does not exist in OpenMPI 2.x, but here oversubscription is allowed by default. – Ettore

Well... Openmpi 3.0 did not exist when OP raised the question. It's good to see that openmpi made oversubscription easier. – Stoush

both work but personally, I would go for this solution – Gilberte