Set LD_LIBRARY_PATH before importing in python

Question

Python uses the PYTHONPATH environment-variable to determine in which folders it should look for modules. You can play around with it by modifying sys.path, which works nicely for pure Python-Modules. But when a module uses shared object files or static libraries, it looks for those in LD_LIBRARY_PATH (on linux), but this can't be changed as easily and is platform dependent as far as I know.

The quick-fix for this problem is of course to set the environment-variable or invoke the script like LD_LIBRARY_PATH=. ./script.py, but then you'll have to set it again for every new shell you open. Also, the .so files in my case will always be in the same directory as the .py file, but may very well be moved to another absolute path, so I'd like to set them automatically every time I invoke the script.

How can I edit the path in which the Python interpreter looks for libraries platform-independently on runtime?

EDIT:

I already tried os.environ['LD_LIBRARY_PATH'] = os.getcwd(), but to no avail.

Uncork

This should probably be handled by the module installer by installing the shared libraries in a standard (though possibly machine-specific) location. – Batty

...using virtualenv :) @Batty – Flyn

@Batty after thinking about it, platform-dependency is something to consider when installing software. Also, windows doesn't use .so or .a but .dll and .lib-files, and my libraries will have to be re-compiled for it one way or another. I just felt like a quick-and-dirty solution would ease testing and development. – Uncork

I removed my answer to a related topic and posted a new question: Trying to import pypyodbc module gives error 'ODBC Library is not found. Is LD_LIBRARY_PATH set?'. – Authorized

Possible duplicate of Changing LD_LIBRARY_PATH at runtime for ctypes – Adventurer

check this: #1178594 – Fireboard

Answer 1

UPDATE: see the EDIT below.

I would use:

import os

os.environ['LD_LIBRARY_PATH'] = os.getcwd()  # or whatever path you want

This sets the LD_LIBRARY_PATH environment variable for the duration/lifetime of the execution of the current process only.

EDIT: it looks like this needs to be set before starting Python: Changing LD_LIBRARY_PATH at runtime for ctypes

So I'd suggest going with a wrapper .sh (or .py if you insist) script. Also, as @chepner pointed out, you might want to consider installing your .so files in a standard location (within the virtualenv).

See also Setting LD_LIBRARY_PATH from inside Python

Flyn

sorry, I should have mentioned that I already tried that and it doesn't work, the import statement after that still throws an ImportError which does not occur if I invoke the script with LD_LIBRARY_PATH=. ./script.py. – Uncork

it's totally unclear to me what the wrapper script should contain. – Floss

Good suggestion for installing the libraries to a standard location within the virtualenv. – Studied

Does this work? I tried this in __init__.py in my compiled pyqt5 to add additional LD_LIBRARY_PATH path to maya 2017 installation, but when I run from PyQt5.QtWidgets import QApplication, QWidget from python interpreter, it still says ImportError: libQt5Widgets.so.5: cannot open shared object file: No such file or directory – Schultz

Try asking a question with Qt, Python, Maya tags – Flyn

Alternatively, you can set LD_LIBRARY_PATH and re-start the Python interpreter using the same command line arguments. See https://mcmap.net/q/48490/-setting-ld_library_path-from-inside-python for details. – Distiller

Answer 2

My solution to this problem is to put this as the first line of a Python script (instead of the usual shebang):

exec env LD_LIBRARY_PATH=/some/path/to/lib /path/to/specific/python -x "$0" "$@"

And here is how this works:

• with no shebang the current shell treats the file as a shell script,
• "exec" ensures that this first line is also the last command from this file executed by the shell,
• "env" is used here to set any environment variables, e.g. LD_LIBRARY_PATH,
• an exact path to Python's interpreter can specified or "env" can find one in PATH,
• "-x" is a Python's option which causes the first line to be ignored by the Python interpreter,
• "$0" is the script name, "$@" is substituted by positional parameters.

Aliped

Answer 3

Python, when gets the values of environment variables as in os.environ['LD_LIBRARY_PATH'] or os.environ['PATH'], it copies the values, into a dictionary, from it's parent process's environment, generally bash (bash process's environment gets carried to the child process, the python running instance).

you can see this environment variable section with env command output from bash.

you can also see/read this env data from /proc/<pid>/environ, by introducing an infinite loop(while 1: pass) after modifying any environment variable.

If you see/read this variable value/data from /proc/<pid>/environ after modifying it inside the python script, you would get to see that the real variable's data doesn't get modified, though the python script shows a modified dictionary key value, updated.

What actually happens when you modify an env variable inside python script, as in os.environ['LD_LIBRARY_PATH']='/<new_location>', is that it just updates the value in local dictionary, which is not mapped to process's env variable section. Hence it won't propagate all the way back to reflect in current process's environment, because ONLY a local dictionary was modified/updated/populated.

Hence if we want to have the new environment variable to be reflected, we should overwrite the memory image of the process with new environment variable data, using execv.

Example:

new_lib = '/<new_location>'
if not new_lib in os.environ['LD_LIBRARY_PATH']:
    os.environ['LD_LIBRARY_PATH'] += ':'+new_lib
    try:
        os.execv(sys.argv[0], sys.argv)
    except Exception as e:
        sys.exit('EXCEPTION: Failed to Execute under modified environment, '+e)

import xyz
#do something else

Limitation: Ideally, python should not allow such modification of os.environ variables. But because there is no constant dictionary data type, it allows modification of the data variable. There is absolutely no use of modifying the values, as it does nothing useful to reflect in running process's real environment, unless execv is used.

Belmonte

regarding this: But because there is no constant dictionary data type < I'm sure if they tried they could place read-only items in a mappingproxy (type(type.__dict__)) in a nt/posix module var imported to os for static env vars, where os.environ could be an envdict instance merging both static and non-static names on __getitem__(), but it's not that big of a deal, so they don't care. ;) – Bosk

perhaps need to find out the actual command line used to execute the program... – Varus

No, this should not be a readonly dict. Modifying os.environ is useful because it is what is passed into subprocesses (which is why your execve trick works). Python does not copy the environment into a dict; it accesses the process's environment via getenv/putenv. Your larger point is still correct: changing the environment doesn't affect the current process. LD_LIBRARY_PATH in particular has already been consulted by the time your python code runs. From man dlopen: If, at the time that the program was started, the environment variable LD_LIBRARY_PATH was defined... – Turco

As @Turco notes, the fact setting LD_LIBRARY_PATH after a process has already started doesn't work relates to the way dlopen works, not to a limitation of os.environ. python3 -c 'import os; os.environ["LD_LIBRARY_PATH"] = os.getcwd(); os.system("echo $LD_LIBRARY_PATH")' prints the current working directory, showing that the current process's environment is getting updated as requested (dlopen just doesn't care since it has already saved the value as it was at process startup). – Berkey

Answer 4

The solution works fine if the env is reinit

import os

os.environ['LD_LIBRARY_PATH'] = os.getcwd()  # or whatever path you want 

The code need to be taken in place....

os.execv(sys.argv[0], sys.argv)

Delivery

Answer 5

Since coreutils 8.30 it is possible to use env -S to split shebang line to separate arguments:

#!/usr/bin/env -S LD_LIBRARY_PATH=/path/to/lib python options

For compatibility with older systems, you can use the fact that the shell allows all commands to be enclosed in quotes, whereas in python it will be just strings:

#!/bin/sh
"export" "LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH"
"exec" "python3" "$0" "$@"
# Further python program
import somemodule

Monogenesis