Distribute a Python package with a compiled dynamic shared library

Question

How do I package a Python module together with a precompiled .so library? Specifically, how do I write setup.py so that when I do this in Python

>>> import top_secret_wrapper

It can easily find top_secret.so without having to set LD_LIBRARY_PATH?

In my module development environment, I have the following file structure:

.
├── top_secret_wrapper
│   ├── top_secret.so
│   └── __init__.py
└── setup.py

Inside __init__.py, I have something like:

import top_secret

Here's my setup.py

from setuptools import setup, Extension

setup(
    name = 'top_secret_wrapper',
    version = '0.1',
    description = 'A Python wrapper for a top secret algorithm',
    url = None,
    author = 'James Bond',
    author_email = '[email protected]',
    license = 'Spy Game License',
    zip_safe = True,
)

I'm sure my setup.py is lacking a setting where I specify the location of top_secret.so, though I'm not sure how to do that.

Floydflss

Answer 1

What I ended up doing is:

setup(
    name='py_my_lib',
    version=version,  # specified elsewhere
    packages=[''],
    package_dir={'': '.'},
    package_data={'': ['py_my_lib.so']},
)

This way I get to import the lib by its name, and don't have another level of nestedness:

import py_my_lib

and not

from py_my_lib_wrapper import py_my_lib

Beitris

I am not sure this is a good solution as at installation it will put the .so directly at pythonx.x/site-packages directory level. IMO a clean package distribution should put every packages files in the package directory i.e.: pythonx.x/site-packages/top_secret/ . – Ioyal

Answer 2

As is mentioned in setupscript.html#installing-package-data:

setup(
    ...
    package_data={'top_secret_wrapper': ['top_secret.so']},
)

Armilla

Answer 3

If that library should also be compiled during install you can describe this as an extension module. If you just want to ship it add it as package_data

Immix

Compiling the .so is not an option, since I do not have the C source for it. – Floydflss

Make it package_data then? – Immix

Answer 4

I managed to bundle a .so (that has other .so dependancies) in its python package directory like this:

• build the mypackage_bindings.cpython-310-x86_64-linux-gnu.so containing Python bindings of C++ using pybind11 and cmake.
• using this minimal python package dir infrastructure:

setup.cfg
setup.py
README.md
mypackage/__init__.py
mypackage/mypackage_bindings.cpython-310-x86_64-linux-gnu.so
mypackage/some_deps.so

• set rpath of mypackage_bindings.so and its .so dependancies to $ORIGIN using these commands on linux (so that the linker will search the deps in the same .so dir):

patchelf --set-rpath '$ORIGIN' mypackage_bindings.cpython-310-x86_64-linux-gnu.so
patchelf --set-rpath '$ORIGIN' some_deps.so

• put in mypackage/__init__.py:

import os
import sys

cur_file_dir = os.path.dirname(os.path.realpath(__file__))

# add current file directory so that mypackage_bindings.so is found by python
sys.path.append(cur_file_dir)

# set current file directory as working dir so that mypackage_bindings.so dependancies
# will be found by the linker (mypackage_bindings.so and its deps RPATH are set to $ORIGIN)
os.chdir(cur_file_dir)

# load every symbols of mypackage_bindings into upper mypackage module
from mypackage_bindings import *

• put in setup.py:

from setuptools import setup

setup(
    name='mypackage',
    packages=['mypackage'],
    package_dir={'mypackage': 'mypackage'},
    package_data={'mypackage': ['*.so', 'lib*']},
    description='Provides mypackage to python users',
    version='0.1',
    url='https://yo.com',
    author='truc muche',
    author_email='[email protected]',
    keywords=['pip', 'mypackage']
    )

• from the minimal python package dir, launch this command to create the python package:

python3 setup.py sdist

That way, there is no need to set LD_LIBRARY_PATH variable, the .so are installed in the pythonX.X/site-packages/mypackage/ directory.

Ioyal

Interesting that you have to set the RPATH for the bindings AND for the dependencies. In principle, RPATH should be propagated through the dependency tree. I am asking this because I noticed some possibly similar issues when using Python bindings (#76532329) and I am wondering if Python is interfering somehow with the RPATH mechanism. – Gabrielagabriele

Are you sure the RPATH is propagated by ld ? I would say that in my given example the 2nd rpath is not necessary. But for my code, as some_deps.so is actually depending on another lib (that I put in the same package dir), I have to set the rpath on some_deps.so. – Ioyal

RPATH is propagated, unless RUNPATH is set. See wiki.debian.org/RpathIssue However, I don't really understand how this works exactly in the context of Python compiled extensions. – Gabrielagabriele

In the link you gave, I assume this line make you think RPATH is propagated : "1. the DT_RPATH dynamic section attribute of the library causing the lookup" ? But for me this does not mean propagation : if liba.so (has RPATH) loads libb.so (does not have RPATH it succeeds). Then, if libb.so tries to load libc.so (does not have RPATH) it fails because ld looks at the library causing the lookup (which is libb.so). – Ioyal

Well, it's true that the link is not clear, but RPATH is propagated, see for instance linux.die.net/man/3/dlopen The fact is that even for If the executable file for the calling program contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag, then the directories listed in the DT_RPATH tag are searched. what the "executable" really is in the context of a Python extension is not really clear to me. – Gabrielagabriele

In python context, the executable is the python interpreter. – Ioyal

That's what I thought too, but that doesn't seem to be what happens in practice. Python interpreters normally available on linux distributions don't have any RPATH or RUNPATH set. On the other hand, RPATH is taken from compiled extensions when imported. RPATH is also propagated through the dependencies of such extensions, but not as it should, or at least not how it does in a pure C/C++ app. – Gabrielagabriele

I did try setting/unsetting RPATH on my library or its sub-library or sub-sub-library and RPATH is not propagated (at least when not set on the executable). If you don't set RPATH on the sub-library, the sub-sub-library is not found when launching python or by inspecting the library using lddtree. – Ioyal

Interesting, I will have to investigate with some toy set-up as soon as I find the time. – Gabrielagabriele