How to include external library with python wheel package

Question

I want to create package for python that embeds and uses an external library (.so) on Linux using the cffi module.

Is there standard way to include .so file into python package?

The package will be used only internally and won't be published to pypi.

I think Wheel packages are the best option - they would create platform specific package with all files ready to be copied so there will be no need to build anything on target environments.

Chiller

Did you try to create the wheel for the package you have? There is good chance, it will include needed file into it (but I am not sure). – Amabil

I tried to build wheel for the package - it included .so file created by cffi, which is linked to external lib, but not the external lib itself. – Chiller

I guess I'll package .so libs into rpm packages (this would also allow me to recompile them from the source if needed) – Chiller

Answer 1

You can use auditwheel to inject the external libraries into the wheel:

auditwheel repair: copies these external shared libraries into the wheel itself, and automatically modifies the appropriate RPATH entries such that these libraries will be picked up at runtime. This accomplishes a similar result as if the libraries had been statically linked without requiring changes to the build system. Packagers are advised that bundling, like static linking, may implicate copyright concerns.

You can pre-build the external c++ library by typically executing the following:

./configure && make && make install

This will generate an my_external_library.so file and install it in the appropriate path. However, you'll need to ensure that the library path is properly set in order for the auditwheel to discover the missing dependency.

export LD_LIBRARY_PATH=/usr/local/lib

You can then build the python wheel by executing:

python setup.py bdist_wheel

Finally, you can repair the wheel, which will inject the my_external_library.so into the package.

auditwheel repair my-python-wheel-1.5.2-cp35-cp35m-linux_x86_64.whl

I successfully applied the above steps to the python library confluent-kafka-python which has a required c/c++ dependency on librdkafka.

---

Note: auditwheel is Linux-only. For MacOS, see the delocate tool.

Herzig

Thanks for a hint. auditwheel comes from pypa (Python Packaging Authority) - they should know what they are doing :) – Chiller

Has anyone tryed it? Is it any good in building arm specific wheels? – Gamut

auditwheel is not for building wheels but for making them manylinux compliant. arm is not a manylinux-compliant platform so it won't help you. but building a non-manylinux wheel that may be linked against some external things is possible with setuptools, for example. – Doenitz

I had to include the argument --plat linux_x86_64 to the auditwheel command, because by default this tool will try to normalize the wheel into manylinux (and since I was doing that from Github Actions, it was crashing because the Docker image used was too recent to make manylinux work correctly) – Unscratched

Answer 2

Wheels are the standard way of distributing Python packages, but there is a problem when you have extension modules that depend on other so's. This is because the normal Linux dynamic linker is used, and that only looks in /usr/lib or /usr/local/lib. This is a problem when installing a wheel in a virtualenv.

As far as I know, you have three options:

• Static linking, so the 'wrapper' does not depend on anything else;
• using ctypes to wrap the so directly from Python;
• Split the distribution into a wheel with the Python code & wrapper, and a separate RPM or DEB to install the so into either /usr/lib or /usr/local/lib.

A wheel may work if you include the dependent so as a data file to be stored in /lib and install into the root Python environment (haven't tried that), but this will break if someone tries to install the wheel into a virtualenv (did try that).

Comorin

Why can't you leave the so files in the package directory under site-packages/? – Disinfectant