Changing LD_LIBRARY_PATH at runtime for ctypes

Question

How do you update this environment variable at runtime so that ctypes can load a library wherever? I've tried the following and neither seem to work.

from ctypes import *
os.environ['LD_LIBRARY_PATH'] = "/home/starlon/Projects/pyCFA635/lib"  
os.putenv('LD_LIBRARY_PATH', "/home/starlon/Projects/pyCFA635/lib")  
lib = CDLL("libevaluator.so")

Kokand

Answer 1

By the time a program such as Python is running, the dynamic loader (ld.so.1 or something similar) has already read LD_LIBRARY_PATH and won't notice any changes thereafter. So, unless the Python software itself evaluates LD_LIBRARY_PATH and uses it to build the possible path name of the library for dlopen() or an equivalent function to use, setting the variable in the script will have no effect.

Given that you say it doesn't work, it seems plausible to suppose that Python does not build and try all the possible library names; it probably relies on LD_LIBRARY_PATH alone.

Urogenous

Answer 2

Even if you give a fully qualified path to CDLL or cdll.LoadLibrary(), you may still need to set LD_LIBRARY_PATH before invoking Python. If the shared library you load explicitly refers to another shared library and no "rpath" is set in the .so for that library, then it won't be found, even if it has already been loaded. An rpath in a library specifies a search path to be used to search for other libraries needed by that library

For example, I have a case of a set of interdependent third-party libraries not produced by me. b.so references a.so. Even if I load a.so in advance:

ctypes.cdll.LoadLibrary('/abs/path/to/a.so')
ctypes.cdll.LoadLibrary('/abs/path/to/b.so')

I get an error on the second load, because b.so refers to simply 'a.so', without an rpath, and so b.so doesn't know that's the correct a.so. So I have to set LD_LIBRARY_PATH in advance to include '/abs/path/to'.

To avoid having to set LD_LIBRARY_PATH, you modify the rpath entry in the .so files. On Linux, there are two utilities I found that do this: chrpath, and patchelf. chrpath is available from the Ubuntu repositories. It cannot change rpath on .so's that never had one. patchelf is more flexible.

Cyclorama

Great tip on using patchelf. patchelf --set-rpath ./ somelib.so did the trick in fixing somelib.so that loads anotherlib.so that resides in the same path. – Heinz

If you're running on Linux and you can link the b.so, one option that solved it for me was adding -Wl,--enable-new-dtags -Wl,-rpath=\$ORIGIN to your linker line such that readelf -d b.so | grep RUNPATH shows (RUNPATH) Library runpath: [$ORIGIN]. This allows searching of the LD_LIBRARY_PATH first but otherwise should "just work" to resolve missing dependencies in the path of your b.so file. – Parget

Alternatively, you can set LD_LIBRARY_PATH and re-start the Python interpreter using the same command line arguments. See https://mcmap.net/q/48490/-setting-ld_library_path-from-inside-python for details. – Delete

Is there any better solution for this use case? – Worm

Answer 3

CDLL can be passed a fully qualified path name, so for example I am using the following in one of my scripts where the .so is in the same directory as the python script.

import os
path = os.path.dirname(os.path.realpath(__file__))
dll = CDLL("%s/iface.so"%path)

In your case the following should suffice.

from ctypes import *
lib = CDLL("/home/starlon/Projects/pyCFA635/lib/libevaluator.so")

Poche

Answer 4

Compile your binary with a rpath relative to the current working directory like:

gcc -shared -o yourbinary.so yoursource.c otherbinary.so \
    -Wl,-rpath='.',-rpath='./another/relative/rpath' -fpic

Then, you are able to change the working directory in python at runtime with:

import os
os.chdir('/path/to/your/binaries')

Like this, the loader also finds other dynamic libraries like otherbinary.so

Likewise

This only works when the working directory of the process loading the library is the directory of the library. Otherwise, the relative runtime path is not able to find the dependency, whose directory is relative to the library. – Helbonnas

c++: error: unrecognized command line option ‘-rpath=.’ – Progestational

Answer 5

Setting LD_LIBRARY_PATH to the path where libraries are placed won't work here and ctypes will not notice any changes. So, you need to set this at the source and run ldconfig before your script to take this into account. Moreover setting the os environment or any PATH variable in the script will have no effect.

I was facing a similar issue and spent around a day to figure this out.

mkdir -p /etc/ld.so.conf.d/
echo "/home/starlon/Projects/pyCFA635/lib" > /etc/ld.so.conf.d/mycustomPath.conf

ldconfig

Then verify if the path is set with:

ldconfig -v | less

After this is done, try to run your script. This has worked for me and should work for you also.

You can see below URL which helped me to resolve this:

https://www.cyberciti.biz/faq/linux-setting-changing-library-path/

Note: I realized the question is old, however I wanted to contribute to this as the accepted answer alone was not actually solving my problem.

Bashee

The method that you proposed requires root privileges. But the user that run the wanted program is not necessary root.... – Springe

@Springe Basically ldconfig need to know the path , So if you are using non-standard custom path , then necessary changes are needed to fix this. – Bashee