Install header-only library with Python

Question

I have a header-only C++ library that I use in my Python extensions. I would like to be able to install them to Python's include path, such that I can compile extensions very easily with python3 setup.py build. I'm partly able, but there are two things that I cannot get working (see below):

1. How can I use python3 setup.py install to install the header files? Currently I only get some *.egg file, but not the headers installed.

2. How can I retain the module's file structure? Currently the file structure is erroneously flattened.

What works

With the following setup.py

from setuptools import setup

setup(
   name        = 'so',
   description = 'Example',
   headers     = [
      'so.h',
   ],
)

I can upload the module to PyPi:

python3 setup.py bdist_wheel --universal
twine upload dist/*

and then install it using pip:

pip3 install so

On my system then I then find the header here

/usr/local/include/python3.6m/so/so.h

which is available when I compile the extensions with Python.

How can I use 'python3 setup.py install'?

Using this strategy I cannot simply run

python3 setup.py install

In that case some so*.egg is installed, but the headers are not stored somewhere where they are available to the compiler.

How to retain a file structure?

When the module is a bit more complicated, and there is some directory hierarchy I also run to problems. For the following setup.py

from setuptools import setup

setup(
  name        = 'so',
  description = 'Example',
  headers     = [
    'so.h',
    'so/implementation.h',
  ],
)

The problem is that the headers are installed to

/usr/local/include/python3.6m/so/so.h
/usr/local/include/python3.6m/so/implementation.h

thus flattening the original file structure.

How can I fix both issues?

Monkfish

Answer 1

How can I use python3 setup.py install to install the header files?

Unfortunately, you can't as long as you're using setuptools. What happens under the hood when you call setuptools.setup()? An egg installer is being built (bdist_egg command) and installed (via easy_install), and neither bdist_egg nor easy_install support including/installing headers. Although the distribution object carries the headers info, it is never requested during the install command. This is an old well-known problem that was never resolved because apparently, installation of header files doesn't fit into the egg build/install procedure.

You thus have three options (or at least three options I know of). Two of them (both inducing a switch to distutils) are not recommended and are provided for completeness sake only:

Bare distutils install (not recommended)

$ sed 's/from setuptools import setup/from distutils.core import setup/' setup.py

This way, the good ol' distutils will take care of the installation when doing python setup.py install, no egg installer will be built and install_headers will be invoked. However, this also includes giving up all the features of setuptools including additional keyword args in setup() and all the other good stuff, needless to say that packages installed via distutils can't be uninstalled with pip.

old-and-unmanageable install (not recommended)

Run the installation with

$ python setup.py install --old-and-unmanageable

This is a switch setuptools provides if you explicitly wish to run the distutils install. The egg installer is not built, instead, the distutils.command.install.install is invoked. Thus, the installation is the same as with bare distutils install.

Drawbacks of this approach: same as with bare distutils install plus: setuptools condemns usage of the switch; if you forget to provide it, you end with installing eggs and have to redo the installation.

Replace python setup.py install with pip install (recommended)

pip is capable of installing packages from source directories; just issue

$ pip install dir/

assuming dir contains the setup.py. This way, a wheel file is built from sources (same as in bdist_wheel; actually, this command is being run first) and installed, managing the installation of header files just fine.

How can I retain the module's file structure?

You will have to tweak the install_headers command a bit:

import os
from distutils.command.install_headers import install_headers as install_headers_orig
from setuptools import setup

class install_headers(install_headers_orig):

    def run(self):
        headers = self.distribution.headers or []
        for header in headers:
            dst = os.path.join(self.install_dir, os.path.dirname(header))
            self.mkpath(dst)
            (out, _) = self.copy_file(header, dst)
            self.outfiles.append(out)

setup(
    name='so',
    headers=['h1.h', 'subtree/h2.h'],
    cmdclass={'install_headers': install_headers},
    ...
)

What is essential here is the line

dst = os.path.join(self.install_dir, os.path.dirname(header))

The vanilla install_headers copies the header files directly to install_dir; the above line in the overloaded install_headers command additionally takes care of the eventual subdirectories in header filenames. When installing the package, the subdirectories should be retained now:

$ pip show -f so | grep include
  ../../../include/site/python3.6/so/h1.h
  ../../../include/site/python3.6/so/subtree/h2.h

Kane

You may also find useful: related issue in Python's bug tracker. I guess nothing changed since 2012... – Kane

Awesome, thanks a lot for you clear answer! Am I correct in observing that the headers can only be in the same folder as setup.py or down from it? E.g. ../src/so.h would not work? Also can I equally use pip to upload the package including the headers to PyPi? How should I do that? – Monkfish

The files may be located outside of the project directory and even lie in some system dir (I actually have to maintain an obscure project that copies headers from /usr/include/). – Kane

But: you have to be careful with relative paths like ../src/so.h because they will be resolved relative to the working directory, and it is not necessarily the one containing setup.py. For example, I can run the script from any other working directory: cd / && python /home/me/myproj/setup.py cmd. You are on the safe side if you either specify the absolute paths, or at least resolve the relative ones against some fixed location, for example __file__ (this would be the path to setup.py script). – Kane

As for your other question - you can build wheels with pip wheel command, but there is nothing for uploading the packages. However, the build process you described it in your question (bdist_wheel + twine upload) is the correct one, I usually use the same command to build and push a package to PyPI. I wouldn't change that. BTW, this is also the recommended way by the PyPA devs, according to these docs. – Kane

I see. I tried resolving it against __file__ (with os.path.join(__file__, name)), but that in my case evaluates to some obscure location: /private/var/folders/11/qf0tz9ws1q39bj4m993xnt0w0000gn/T/pip-req-build-gz4u8q21/setup.py/../src/so.h, which is not the folder where my library is in?? – Monkfish

Hmm, indeed pip install . copies the whole source dir to temp and builds from there, so all relative paths are broken even relative to __file__. I guess you have to work with absolute paths then. If you don't like the hardcoded absolute include path in your setup script, you could at least make it configurable - maybe passing the include root in an environment variable or store it in a config file? – Kane

If there are no robust options indeed it is really a matter of taste. I guess I can slightly change the file-structure for this project. Thanks once again! – Monkfish

Glad I could help! – Kane

distutils is deprecated. See PEP632. This must not be used. What directive must I use to include header files in the tar.gz file produced by python3 -m build ? – Pieper

@Pieper I have not recommended distutils in the answer long before it was deprecated ;-) as for your question, you can't, as setuptools doesn't support passing headers via setup.cfg. – Kane

@Kane you are right that we can't with setuptools. In the mean time I found out that python3 -m build is using sdist to create the tar.gz file. It won't include header files by default. A MANIFEST.in file is required. Thank you for taking the time to reply to my question. – Pieper