Loading shared library in open-mpi/ mpi-run

Question

I'm trying to run my program using torque scheduler using mpi run. Though in my pbs file I load all the library by

export LD_LIBRARY_PATH=/path/to/library

yet it gives error i.e.

error while loading shared libraries: libarmadillo.so.3: 
cannot open shared object file: No such file or directory. 

I guess error lies in variable LD_LIBRARY_PATH not set in all the nodes. How would I make it work?

Practitioner

Could you post your PBS script here? – Durwin

Answer 1

LD_LIBRARY_PATH is not exported automatically to MPI processes, spawned by mpirun. You should use

mpirun -x LD_LIBRARY_PATH ...

to push the value of LD_LIBRARY_PATH. Also make sure that the specified path exists on all nodes in the cluster and that libarmadillo.so.3 is available everywhere.

Filtration

If Open MPI was compiled with native Torque/TM support, the LD_LIBRARY_PATH (and the rest of the environment) should be propagated out to all the nodes in the allocation automatically. If Open MPI wasn't built with the native Torque support, then I'm guessing you're using rsh/ssh to launch, in which case the "-x LD_LIBRARY_PATH" clause is necessary. I'd actually advise re-building OMPI with Torque/TM support; it's better than rsh/ssh support for multiple reasons. – Premise

THANK YOU. I WAS DRIVING MAD WITH THIS SHIT.... Exporting the LD_LIBRARY_PATH is supposed to let child processes use those variables, but it turns out it doesn't. – Intercession

@GregK. exporting only works for processes launched on the same node where mpiexec runs as those are child processes of the latter. Processes launched on remote nodes are children of the respective ORTE shepherd daemon and don't directly inherit the environment of mpiexec but only the environment that gets forwarded to and additionally set by the daemon. (I might be wrong; Jeff Squyres knows better) – Filtration

Answer 2

On some systems, your environment isn't always propagated via mpirun. You should set all those variables in your .bashrc file.

Brantbrantford