Setting LD_LIBRARY_PATH from inside Python

Question

Is there a way to set specify during runtime where Python looks for shared libraries?

I have fontforge.so located in fontforge_bin and tried the following

os.environ['LD_LIBRARY_PATH']='fontforge_bin'
sys.path.append('fontforge_bin')
import fontforge

and get

ImportError: fontforge_bin/fontforge.so: cannot open shared object file: No such file or directory

Doing ldd on fontforge_bin/fontforge.so gives the following

linux-vdso.so.1 =>  (0x00007fff2050c000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f10ffdef000)
libc.so.6 => /lib/libc.so.6 (0x00007f10ffa6c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f110022d000)

Durward

This works if I do export LD_LIBRARY_PATH=fontforge_bin before running the script, now I want to set this from inside the script – Durward

Using sys.path.append was the right way, and as you can see from the error message it did try to open it. I suggest that you use the full path name of the directory, rather than a relative one. – Isotropic

Answer 1

Your script can check for the existence/properness of the environment variable before you import your module, then set it in os.environ if it is missing, and then call os.execv() to restart the python interpreter using the same command line arguments but an updated set of environment variables.

This is only advisable before any other imports (other than os and sys), because of potential module-import side-effects, like opened file descriptors or sockets, which may be challenging to close cleanly.

This code sets LD_LIBRARY_PATH and ORACLE_HOME:

#!/usr/bin/python
import os, sys
if 'LD_LIBRARY_PATH' not in os.environ:
    os.environ['LD_LIBRARY_PATH'] = '/usr/lib/oracle/XX.Y/client64/lib'
    os.environ['ORACLE_HOME'] = '/usr/lib/oracle/XX.Y/client64'
    try:
        os.execv(sys.argv[0], sys.argv)
    except Exception, exc:
        print 'Failed re-exec:', exc
        sys.exit(1)
#
# import yourmodule
print 'Success:', os.environ['LD_LIBRARY_PATH']
# your program goes here

It's probably cleaner to set that environment variable as part of the starting environment (in the parent process or systemd/etc job file).

Annals

This probably only works on linux and other unix variants. I don't know if it will work on Windows or Darwin. – Annals

It looks that apache only accept LD_LIBRARY_PATH from /etc/sysconfig/httpd. The way mentioned above didn't work for my case. – Costermansville

Don't try this within reticulate. – Serieswound

And if 'LD_LIBRARY_PATH' in os.environ? – Little

This may cause an infinite loop! – Superfine

Answer 2

...well sort of you could load all libraries from some folder of your choosing via ctypes and thus make them available for you regardless of the LD_LIBRARY_PATH.

from ctypes import *
lib1 = cdll.LoadLibrary('/home/username/lib/some_library.so')

or iterate through the files in that dir... you get the idea, once it is loaded it is there for you [if the dependencies are also out of the default path you should load them too...].

Eutectoid

Didn't work for me. The library still cannot be found when subsequently importing the module that uses it. – Yearwood

I confirm this solution worked in a scenario very similar to the one described in the question. In Linux (Ubuntu 16.04). Needed to import native Python module foo.so that depends on libfoo.so in the same local path. First I tried setting RPATH to $ORIGIN/. in foo.so. It works only if I ran the script in the same directory. – Confessional

Just to keep in mind, don't load 64bit dll/so files in 32bit python and vice versa. – Eutectoid

This does not work if your so file has further dependencies.... – Finnic

This worked for me for lightgbm...used: cdll.LoadLibrary('/mnt/data-science/miniconda3/lib/libgomp.so.1') – Vermiculate

Answer 3

LD_LIBRARY_PATH sets the dynamic linker path; that generally can't be changed at runtime, since it's usually cached by the dynamic linker.

That's not where Python looks for imports, though, including module imports. Changing sys.path is correct.

# ls foo/
_csv.so
# python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
>>> import sys
>>> sys.path.insert(0, "foo")
>>> import _csv
>>> _csv.__file__
'foo/_csv.so'

(By the way, you may want to ldd the library to see if you have any odd import paths in the library. "ImportError: fontforge_bin/fontforge.so" looks strange.)

Jural

I added ldd output to my question, not sure if it qualifies as odd. But there are 3 required libraries inside fontforge_bin that I don't think the linker would find. I found another case of strange linker error message -- when I try to load 64-bit "fontforge.so" under 32-bit Python, I get "file not found" when it tries to load "fontforge.so" – Durward