How to build and distribute a Python/Cython package that depends on third party libFoo.so

Asked 31/10, 2017 at 18:41 Answered 10/11, 2017 at 19:47

Solved python setuptools python-extensions python-packaging

I've written a Python module that depends on some C extensions. Those C extensions depend in turn on several compiled C libraries. I'd like to be able to distribute this module bundled with all the dependencies.

I've put together a minimal example (it can be found on GitHub in its entirety).

The directory structure is:

$ tree .
.
├── README.md
├── poc
│   ├── __init__.py
│   ├── cython_extensions
│   │   ├── __init__.py
│   │   ├── cvRoberts_dns.c
│   │   ├── cvRoberts_dns.h
│   │   ├── helloworld.c
│   │   ├── helloworld.pxd
│   │   ├── helloworld.pyx
│   │   ├── test.c
│   │   └── test.h
│   ├── do_stuff.c
│   └── do_stuff.pyx
└── setup.py

setup.py builds the extensions, and links against the necessary libraries (libsundials_cvode, libsundials_nvectorserial in this case):

from setuptools import setup, find_packages
from setuptools.extension import Extension
from Cython.Build import cythonize


ext_module_dostuff = Extension(
    'poc.do_stuff',
    ['poc/do_stuff.pyx'],
)

ext_module_helloworld = Extension(
    'poc.cython_extensions.helloworld',
    ['poc/cython_extensions/helloworld.pyx', 'poc/cython_extensions/test.c', 'poc/cython_extensions/cvRoberts_dns.c'],
    include_dirs = ['/usr/local/include'],
    libraries = ['m', 'sundials_cvodes', 'sundials_nvecserial'],
    library_dirs = ['/usr/local/lib'],
)

cython_ext_modules = [
   ext_module_dostuff,
   ext_module_helloworld
]


setup (
  name = "poc",
  ext_modules = cythonize(cython_ext_modules),
  packages=['poc', 'poc.cython_extensions'],
)

This is all well and good, but it does require that the end user first install sundials (and, in the actual case, several other libraries that are extremely finicky to get up and running).

Ideally, I'd like to be able to set this up only on development machines, create a distribution that includes the appropriate shared libraries, and ship some sort of bundle.

Given the various tutorials, examples and SO posts I've found so far. I'm led to believe I'm on the right track. However, there's some sort of final step that I'm just not groking.

Any help is appreciated :-).

Jessjessa answered 31/10, 2017 at 18:41 Comment(3)

For what platform do you want to achieve this? – Mcquade 6/11, 2017 at 16:21

I want to be able to distribute to OSX and Ubuntu. If cross-compiling/-distributing is difficult, I can get two dedicated development machines. – Jessjessa 6/11, 2017 at 20:21

If your dependency is compiled well for the platform then it wont be a bad idea i believe. What you need is to use #16247423 and #46726776. so basically you want to define extra_link_args=['-lsundials_cvodes -lsundials_nvecserial -static'] in your setup.py and then build. This should build the dependency also. If that doesn't work then let me know. You might have to tweak the flags as shown in first thread – Magnesia 6/11, 2017 at 20:50

As you probably know, the recommended way of distributing a Python module with compiled components is to use the wheel format. There doesn't appear to be any standard cross-platform way of bundling third-party native libraries into a wheel. However, there are platform-specific tools for this purpose.

On Linux, use `auditwheel`.

auditwheel modifies an existing Linux wheel file to add any third-party libraries which are not included in the basic "manylinux" standard. Here's an walkthrough of how to use it with your project on a clean install of Ubuntu 17.10:

First, install basic Python development tools, and the third-party library with its headers:

root@ubuntu-17:~# apt-get install cython python-pip unzip
root@ubuntu-17:~# apt-get install libsundials-serial-dev

Then build your project into a wheel file:

root@ubuntu-17:~# cd cython-example/
root@ubuntu-17:~/cython-example# python setup.py bdist_wheel
[...]
root@ubuntu-17:~/cython-example# cd dist/
root@ubuntu-17:~/cython-example/dist# ll
total 80
drwxr-xr-x 2 root root  4096 Nov  8 11:28 ./
drwxr-xr-x 7 root root  4096 Nov  8 11:28 ../
-rw-r--r-- 1 root root 70135 Nov  8 11:28 poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Archive:  poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   116648  2017-11-08 11:28   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
      105  2017-11-08 11:28   poc-0.0.0.dist-info/WHEEL
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
      793  2017-11-08 11:28   poc-0.0.0.dist-info/RECORD
---------                     -------
   180382                     10 files

The wheel file can now be installed locally and tested:

root@ubuntu-17:~/cython-example/dist# pip install poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

Now we install the auditwheel tool. It requires Python 3, but it's capable of processing wheels for Python 2 or 3.

root@ubuntu-17:~/cython-example/dist# apt-get install python3-pip
root@ubuntu-17:~/cython-example/dist# pip3 install auditwheel

auditwheel uses another tool called patchelf to do its job. Unfortunately, the version of patchelf included with Ubuntu 17.10 is missing a bugfix without which auditwheel will not work. So we'll have to build it from source (script taken from the manylinux Docker image):

root@ubuntu-17:~# apt-get install autoconf
root@ubuntu-17:~# PATCHELF_VERSION=6bfcafbba8d89e44f9ac9582493b4f27d9d8c369
root@ubuntu-17:~# curl -sL -o patchelf.tar.gz https://github.com/NixOS/patchelf/archive/$PATCHELF_VERSION.tar.gz
root@ubuntu-17:~# tar -xzf patchelf.tar.gz
root@ubuntu-17:~# (cd patchelf-$PATCHELF_VERSION && ./bootstrap.sh && ./configure && make && make install)

Now we can check which third-party libraries the wheel requires:

root@ubuntu-17:~/cython-example/dist# auditwheel show poc-0.0.0-cp27-cp27mu-linux_x86_64.whl

poc-0.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.4',
'GLIBC_2.2.5', 'GLIBC_2.3.4'}

The following external shared libraries are required by the wheel:
{
    "libblas.so.3": "/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1",
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.26.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libgfortran.so.4": "/usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0",
    "liblapack.so.3": "/usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.26.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.26.so",
    "libquadmath.so.0": "/usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0",
    "libsundials_cvodes.so.2": "/usr/lib/libsundials_cvodes.so.2.0.0",
    "libsundials_nvecserial.so.0": "/usr/lib/libsundials_nvecserial.so.0.0.2"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libblas.so.3, libgfortran.so.4, liblapack.so.3, libquadmath.so.0,
libsundials_cvodes.so.2, libsundials_nvecserial.so.0

And create a new wheel which bundles them:

root@ubuntu-17:~/cython-example/dist# auditwheel repair poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Repairing poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/libsundials_nvecserial.so.0.0.2 -> poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
Grafting: /usr/lib/libsundials_cvodes.so.2.0.0 -> poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
Grafting: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 -> poc/.libs/liblapack-549933c4.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 -> poc/.libs/libblas-52fa99c8.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0 -> poc/.libs/libgfortran-2df4b07d.so.4.0.0
Grafting: /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0 -> poc/.libs/libquadmath-0d7c3070.so.0.0.0
Setting RPATH: poc/cython_extensions/helloworld.so to "$ORIGIN/../.libs"
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64

Fixed-up wheel written to /root/cython-example/dist/wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Archive:  wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
     1400  2017-11-08 12:08   poc-0.0.0.dist-info/RECORD
      110  2017-11-08 12:08   poc-0.0.0.dist-info/WHEEL
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   131712  2017-11-08 12:08   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
   230744  2017-11-08 12:08   poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
  7005072  2017-11-08 12:08   poc/.libs/liblapack-549933c4.so.3.7.1
   264024  2017-11-08 12:08   poc/.libs/libquadmath-0d7c3070.so.0.0.0
  2039960  2017-11-08 12:08   poc/.libs/libgfortran-2df4b07d.so.4.0.0
    17736  2017-11-08 12:08   poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
   452432  2017-11-08 12:08   poc/.libs/libblas-52fa99c8.so.3.7.1
---------                     -------
 10206026                     16 files

If we uninstall the third-party libraries, the previously-installed wheel will stop working:

root@ubuntu-17:~/cython-example/dist# apt-get remove libsundials-serial-dev && apt-get autoremove
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "poc/do_stuff.pyx", line 1, in init poc.do_stuff
ImportError: libsundials_cvodes.so.2: cannot open shared object file: No such file or directory

But the wheel with the bundled libraries will work fine:

root@ubuntu-17:~/cython-example/dist# pip uninstall poc
[...]
root@ubuntu-17:~/cython-example/dist# pip install wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

On OSX, use `delocate`.

delocate for OSX apparently works very similarly to auditwheel. Unfortunately I don't have an OSX machine available to provide a walkthrough.

Combined example:

One project which uses both tools is SciPy. This repository, despite its name, contains the official SciPy build process for all platforms, not just Mac. Specifically, compare the Linux build script (which uses auditwheel), with the OSX build script (which uses delocate).

To see the result of this process, you might want to download and unzip some of the SciPy wheels from PyPI. For example, scipy-1.0.0-cp27-cp27m-manylinux1_x86_64.whl contains the following:

 38513408  2017-10-25 06:02   scipy/.libs/libopenblasp-r0-39a31c03.2.18.so
  1023960  2017-10-25 06:02   scipy/.libs/libgfortran-ed201abd.so.3.0.0

While scipy-1.0.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl contains this:

   273072  2017-10-25 07:03   scipy/.dylibs/libgcc_s.1.dylib
  1550456  2017-10-25 07:03   scipy/.dylibs/libgfortran.3.dylib
   279932  2017-10-25 07:03   scipy/.dylibs/libquadmath.0.dylib

Abstraction answered 8/11, 2017 at 15:1 Comment(1)

This looks awesome. I should be able to try this out tonight. Thanks. – Jessjessa 9/11, 2017 at 18:53

To enhance mhsmith's excellent answer, here are the steps performed on MacOS with delocate:

Install sundials, for example with Homebrew:
```
$ brew install sundials
```
Build the package:
```
$ python setup.py bdist_wheel
```

The pendants to auditwheel show/auditwheel repair are delocate-listdeps/delocate-wheel, so first analyze the resulting wheel file:

$ delocate-listdeps --all dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
/usr/lib/libSystem.B.dylib
/usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib
/usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib

Fixing the wheel file:

$ delocate-wheel -v -w dist_fixed dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
Fixing: dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
Copied to package .dylibs directory:
  /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib
  /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib

In the dist_fixed directory, you will have the bundled wheel. You will notice the size difference:

$ ls -l dist/ dist_fixed/
dist/:
total 72
-rw-r--r--  1 hoefling  wheel  36030 10 Nov 20:25 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

dist_fixed/:
total 240
-rw-r--r--  1 hoefling  wheel  120101 10 Nov 20:34 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

If you list deps for the bundled wheel, you will notice the needed libraries are now bundled (indicated by prefix @loader_path):

$ delocate-listdeps --all dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
/usr/lib/libSystem.B.dylib
@loader_path/../.dylibs/libsundials_cvodes.2.9.0.dylib
@loader_path/../.dylibs/libsundials_nvecserial.2.7.0.dylib

Installing the bundled wheel (notice the bundled libs are installed correctly):

$ pip install dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
Processing ./dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
Installing collected packages: poc
Successfully installed poc-0.0.0
$ pip show -f poc
Name: poc
Version: 0.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /Users/hoefling/.virtualenvs/stackoverflow-py27/lib/python2.7/site-packages
Requires: 
Files:
  poc-0.0.0.dist-info/DESCRIPTION.rst
  poc-0.0.0.dist-info/INSTALLER
  poc-0.0.0.dist-info/METADATA
  poc-0.0.0.dist-info/RECORD
  poc-0.0.0.dist-info/WHEEL
  poc-0.0.0.dist-info/metadata.json
  poc-0.0.0.dist-info/top_level.txt
  poc/.dylibs/libsundials_cvodes.2.9.0.dylib
  poc/.dylibs/libsundials_nvecserial.2.7.0.dylib
  poc/__init__.py
  poc/__init__.pyc
  poc/cython_extensions/__init__.py
  poc/cython_extensions/__init__.pyc
  poc/cython_extensions/helloworld.so
  poc/do_stuff.so

Obturate answered 10/11, 2017 at 19:47 Comment(0)

I'd suggest to take a completely different approach. Set up a Linux package management infrastructure. On Ubuntu/Debian this could be done with reprepro. https://wiki.ubuntuusers.de/reprepro/ could be a start, but there are much more tutorials available. You could then build your own Linux package distributing your libraries and all necessary files together with your Python application.

This would be a very clean and convenient approach for your clients. Especially regarding updates. (You even can address different OS releases as necessary at the same time.)

As always a clean approach comes with a cost. This clean approach takes quite some effort for you to implement. You need to not only set up a server - that's the easier part - but get into how to build packages - which is not difficult but you'll be required to read a bit how to do that and do quite a bit of experimentation to end up with packages being exactly as you want them. However everything WILL be the way you want it then. And future updates are really easy for you as well as your client machines.

I'd recommend that approach iff you want to simplify updates in the future, want to learn about Linux and might have requirements for own packages in the future. Or a large amount of clients.

That about a very "high level" approach. In contrast very "low level" approach would be the following one:

Check the presence of your libraries on startup of your program
If not present: Terminate the application. Print a text that refers to a script how to install the necessary libraries. That could even be an URL where to download the script, f.e. with:

bash <(curl -s http://mywebsite.com/myscript.txt)

Sedge answered 10/11, 2017 at 6:25 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

On Linux, use auditwheel.

On OSX, use delocate.

Combined example:

Recommended topics

Hot tags

On Linux, use `auditwheel`.

On OSX, use `delocate`.