How to set environment variables on compute nodes in an MPI job

Question

I don't understand how the environment is set on compute nodes when running with MPI under a scheduler.

I do:

mpirun -np 1 --hostfile ./hostfile foo.sh

with foo.sh:

#!/usr/bin/env zsh                                                                                                  
echo $LD_LIBRARY_PATH

Then I do not recover the LD_LIBRARY_PATH I have got in an interactive shell... What are the initialization files that are executed/sourced at connection with MPI?

note: I am under zsh, and I tried to put things in .zprofile or .zshenv instead of .zshrc, but it doesn't seem to make a change... My LD_LIBRARY_PATH is set in a .profile which is sourced by a .bashrc which is sourced by the .zshrc.

Kudu

how do you expect sh to read your zsh files? – Mehetabel

My .zshrc includes a .bashrc which includes a .profile in which the LD_LIBRARY_PATH is set. I edited the shebang with zsh to avoid confusion, but it does not make any difference – Kudu

Do run run mpirun -np 1 --hostfile ./hostfile echo $LD_LIBRARY_PATH or mpirun -np 1 --hostfile ./hostfile 'echo $LD_LIBRARY_PATH'? In the first case, the variable is expanded before mpirun is even called. – Dielectric

Yes of course you are right, what an idiot am I... But it does not solve my problem to have a good LD_LIBRARY_PATH in the mpi-shells. Besides, mpirun -np 1 --hostfile ./hostfile 'echo $LD_LIBRARY_PATH' does not work for me: mpirun was unable to launch the specified application as it could not find an executable:..Executable: echo $LD_LIBRARY_PATH – Kudu

Answer 1

Some MPI implementations have an -x flag for mpirun for this, e.g. OpenMPI:

-x <env>

Export the specified environment variables to the remote nodes before executing the program. Only one environment variable can be specified per -x option. Existing environment variables can be specified or new variable names specified with corresponding values. For example:

% mpirun -x DISPLAY -x OFILE=/tmp/out ...

The parser for the -x option is not very sophisticated; it does not even understand quoted values. Users are advised to set variables in the environment, and then use -x to export (not define) them.

If your's does not, you'll have to explicitly set the environment variables in your job script, e.g.

export LD_LIBRARY_PATH=...

Dielectric

I knew this option -x (which is able to solves my problem), but don't understand why I have to use it. I though MPI use some kind of ssh to log on some hosts, and so sources the .profile or other initilization files... – Kudu

It can be setup to do that, but it doesn't have to. The Remote Execution paragaph in the manpage I linked to in my answer has a bit to say on this. – Dielectric

This is the right approach. MPI runs even on machines where there is nothing like a shell on the compute nodes (e.g., Blue Gene); there's certainly no guarantee that mpirun ssh-es into each machine and starts a shell with the appropriate environment. Implementations on clusters often will do that for small node #s, but even then, not if something faster is available (like tm on systems running Torque). So either using an implementation-specific option to export environment variables as above, or wrapping your executable in a script that sets environment variables are the usual methods. – Peterson

Answer 2

You could specify the number of threads per mpi using end with following command as well.

env OMP_NUM_THREADS=n PARALLEL=n mpirun -np m program.exe < input.file > output.file &

where n and m are the number of threads and number of CPU cores.

Example:

env OMP_NUM_THREADS=2 PARALLEL=2 mpirun -np 12 program.exe < input.file > output.file &

Authenticity