How should I structure a Python package that contains Cython code

Question

I'd like to make a Python package containing some Cython code. I've got the the Cython code working nicely. However, now I want to know how best to package it.

For most people who just want to install the package, I'd like to include the .c file that Cython creates, and arrange for setup.py to compile that to produce the module. Then the user doesn't need Cython installed in order to install the package.

But for people who may want to modify the package, I'd also like to provide the Cython .pyx files, and somehow also allow for setup.py to build them using Cython (so those users would need Cython installed).

How should I structure the files in the package to cater for both these scenarios?

The Cython documentation gives a little guidance. But it doesn't say how to make a single setup.py that handles both the with/without Cython cases.

Sanjak

I see the question is getting more up-votes than any of the answers. I'm curious to know why people may find the answers unsatisfactory. – Sanjak

I found this section of the documentation, which gives the answer exactly. – Shiprigged

Answer 1

I've done this myself now, in a Python package simplerandom (BitBucket repo - EDIT: now github) (I don't expect this to be a popular package, but it was a good chance to learn Cython).

This method relies on the fact that building a .pyx file with Cython.Distutils.build_ext (at least with Cython version 0.14) always seems to create a .c file in the same directory as the source .pyx file.

Here is a cut-down version of setup.py which I hope shows the essentials:

from distutils.core import setup
from distutils.extension import Extension

try:
    from Cython.Distutils import build_ext
except ImportError:
    use_cython = False
else:
    use_cython = True

cmdclass = {}
ext_modules = []

if use_cython:
    ext_modules += [
        Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.pyx"]),
    ]
    cmdclass.update({'build_ext': build_ext})
else:
    ext_modules += [
        Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.c"]),
    ]

setup(
    name='mypackage',
    ...
    cmdclass=cmdclass,
    ext_modules=ext_modules,
    ...
)

I also edited MANIFEST.in to ensure that mycythonmodule.c is included in a source distribution (a source distribution that is created with python setup.py sdist):

...
recursive-include cython *
...

I don't commit mycythonmodule.c to version control 'trunk' (or 'default' for Mercurial). When I make a release, I need to remember to do a python setup.py build_ext first, to ensure that mycythonmodule.c is present and up-to-date for the source code distribution. I also make a release branch, and commit the C file into the branch. That way I have a historical record of the C file that was distributed with that release.

Sanjak

Thanks, this is exactly what I needed for a Pyrex project I'm opening up! The MANIFEST.in tripped me up for a second, but I just needed that one line. I'm including the C file in source control out of interest, but I see your point that it's unnecessary. – Wilscam

I've edited my answer to explain how the C file is not in trunk/default, but is added to a release branch. – Sanjak

@CraigMcQueen thanks for the great answer, it helped me a lot! I am wondering however, is it desired behavior to use Cython when available? It seems to me that it would be better to by default use pre-generated c files, unless user explicitly wants to use Cython, in which case he can set the environment variable or something. That would make installation more stable/robust, because user may get different results based on which version of Cython he has installed - he may not even be aware that he has it installed and that it is affecting the building of package. – Saxton

Answer 2

Adding to Craig McQueen's answer: see below for how to override the sdist command to have Cython automatically compile your source files before creating a source distribution.

That way your run no risk of accidentally distributing outdated C sources. It also helps in the case where you have limited control over the distribution process e.g. when automatically creating distributions from continuous integration etc.

from distutils.command.sdist import sdist as _sdist

...

class sdist(_sdist):
    def run(self):
        # Make sure the compiled Cython files in the distribution are up-to-date
        from Cython.Build import cythonize
        cythonize(['cython/mycythonmodule.pyx'])
        _sdist.run(self)
cmdclass['sdist'] = sdist

Suspensive

Answer 3

http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules

It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.

It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he probably doesn’t want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.

This simply means that the setup.py file that you ship with will just be a normal distutils file on the generated .c files, for the basic example we would have instead:

from distutils.core import setup
from distutils.extension import Extension
 
setup(
    ext_modules = [Extension("example", ["example.c"])]
)

Sedgewake

Answer 4

The easiest is to include both but just use the c-file? Including the .pyx file is nice, but it's not needed once you have the .c file anyway. People who want to recompile the .pyx can install Pyrex and do it manually.

Otherwise you need to have a custom build_ext command for distutils that builds the C file first. Cython already includes one. http://docs.cython.org/src/userguide/source_files_and_compilation.html

What that documentation doesn't do is say how to make this conditional, but

try:
     from Cython.distutils import build_ext
except ImportError:
     from distutils.command import build_ext

Should handle it.

Squish

Thanks for your answer. That's reasonable, although I prefer if the setup.py can build directly from the .pyx file when Cython is installed. My answer has implemented that as well. – Sanjak

Well, that's the whole point of my answer. It was just not a complete setup.py. – Squish

Answer 5

Including (Cython) generated .c files are pretty weird. Especially when we include that in git. I'd prefer to use setuptools_cython. When Cython is not available, it will build an egg which has built-in Cython environment, and then build your code using the egg.

A possible example: https://github.com/douban/greenify/blob/master/setup.py

---

Update(2017-01-05):

Since setuptools 18.0, there's no need to use setuptools_cython. Here is an example to build Cython project from scratch without setuptools_cython.

Dampen

does this fix the issue of Cython not being installed even though you specify it in setup_requires? – Hearsay

also isn't possible to put 'setuptools>=18.0' in setup_requires instead of creating the method is_installed? – Hearsay

@capitalistpug First you need to make sure setuptools>=18.0 has been installed, then you only need to put 'Cython >= 0.18' in setup_requires, and Cython will be installed during install progress. But if you're using setuptools < 18.0, even you specific cython in setup_requires, it will not be installed, in this case, you should consider use setuptools_cython. – Dampen

Thanks @McKelvin, this seems as a great solution! Is there any reason why should we use the other approach, with cythonizing the source files in advance, next to this? I tried your approach and it does seem to be somewhat slow when installing (takes a minute to install but builds in a second). – Saxton

@Saxton If you find it's slow. There might be 2 reasons: 1. It's slow to install the Cython since it needs to build Cython from source first. In this case, try install pythonwheels.com first. Try again and a binary of Cython will be installed which is very quick. 2. The "cythonize" command allows for multi-threaded compilation but we're not able to use it in this way so source files are compiled one by one which might be slow. – Dampen

Thanks @McKelvin - what do you mean by install pythonwheels.com? Setuptools use easy install if I am correct, and I do not know a way to change that. I have only one source file for Cython to compile. – Saxton

@Saxton pip install wheel. Then it must be reason 1. Please install wheel first and try again. – Dampen

Answer 6

All other answers either rely on

• distutils
• importing from Cython.Build, which creates a chicken-and-egg problem between requiring cython via setup_requires and importing it.

A modern solution is to use setuptools instead, see this answer (automatic handling of Cython extensions requires setuptools 18.0, i.e., it's available for many years already). A modern standard setup.py with requirements handling, an entry point, and a cython module could look like this:

from setuptools import setup, Extension

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name='MyPackage',
    install_requires=requirements,
    setup_requires=[
        'setuptools>=18.0',  # automatically handles Cython extensions
        'cython>=0.28.4',
    ],
    entry_points={
        'console_scripts': [
            'mymain = mypackage.main:main',
        ],
    },
    ext_modules=[
        Extension(
            'mypackage.my_cython_module',
            sources=['mypackage/my_cython_module.pyx'],
        ),
    ],
)

Meier

Importing from Cython.Build at setup time causes ImportError for me. Having setuptools to compile pyx is the best way to do it. – Reduction

Answer 7

The simple hack I came up with:

from distutils.core import setup

try:
    from Cython.Build import cythonize
except ImportError:
    from pip import pip

    pip.main(['install', 'cython'])

    from Cython.Build import cythonize


setup(…)

Just install Cython if it could not be imported. One should probably not share this code, but for my own dependencies it's good enough.

Varicella

Answer 8

This is a setup script I wrote which makes it easier to include nested directories inside the build. One needs to run it from folder within a package.

Givig structure like this:

__init__.py
setup.py
test.py
subdir/
      __init__.py
      anothertest.py

setup.py

from setuptools import setup, Extension
from Cython.Distutils import build_ext
# from os import path
ext_names = (
    'test',
    'subdir.anothertest',       
) 

cmdclass = {'build_ext': build_ext}
# for modules in main dir      
ext_modules = [
    Extension(
        ext,
        [ext + ".py"],            
    ) 
    for ext in ext_names if ext.find('.') < 0] 
# for modules in subdir ONLY ONE LEVEL DOWN!! 
# modify it if you need more !!!
ext_modules += [
    Extension(
        ext,
        ["/".join(ext.split('.')) + ".py"],     
    )
    for ext in ext_names if ext.find('.') > 0]

setup(
    name='name',
    ext_modules=ext_modules,
    cmdclass=cmdclass,
    packages=["base", "base.subdir"],
)
#  Build --------------------------
#  python setup.py build_ext --inplace

Happy compiling ;)

Breastplate

Answer 9

The easiest way I found using only setuptools instead of the feature limited distutils is

from setuptools import setup
from setuptools.extension import Extension
try:
    from Cython.Build import cythonize
except ImportError:
    use_cython = False
else:
    use_cython = True

ext_modules = []
if use_cython:
    ext_modules += cythonize('package/cython_module.pyx')
else:
    ext_modules += [Extension('package.cython_module',
                              ['package/cython_modules.c'])]

setup(name='package_name', ext_modules=ext_modules)

Gilroy

In fact, with setuptools there is no need for the explicit try/catched import from Cython.Build, see my answer. – Meier

Answer 10

I think I found a pretty good way of doing this by providing a custom build_ext command. The idea is the following:

1. I add the numpy headers by overriding finalize_options() and doing import numpy in the body of the function, which nicely avoids the problem of numpy not being available before setup() installs it.

2. If cython is available on the system, it hooks into the command's check_extensions_list() method and by cythonizes all out-of-date cython modules, replacing them with C extensions that can later handled by the build_extension() method. We just provide the latter part of the functionality in our module too: this means that if cython is not available but we have a C extension present, it still works, which allows you to do source distributions.

Here's the code:

import re, sys, os.path
from distutils import dep_util, log
from setuptools.command.build_ext import build_ext

try:
    import Cython.Build
    HAVE_CYTHON = True
except ImportError:
    HAVE_CYTHON = False

class BuildExtWithNumpy(build_ext):
    def check_cython(self, ext):
        c_sources = []
        for fname in ext.sources:
            cname, matches = re.subn(r"(?i)\.pyx$", ".c", fname, 1)
            c_sources.append(cname)
            if matches and dep_util.newer(fname, cname):
                if HAVE_CYTHON:
                    return ext
                raise RuntimeError("Cython and C module unavailable")
        ext.sources = c_sources
        return ext

    def check_extensions_list(self, extensions):
        extensions = [self.check_cython(ext) for ext in extensions]
        return build_ext.check_extensions_list(self, extensions)

    def finalize_options(self):
        import numpy as np
        build_ext.finalize_options(self)
        self.include_dirs.append(np.get_include())

This allows one to just write the setup() arguments without worrying about imports and whether one has cython available:

setup(
    # ...
    ext_modules=[Extension("_my_fast_thing", ["src/_my_fast_thing.pyx"])],
    setup_requires=['numpy'],
    cmdclass={'build_ext': BuildExtWithNumpy}
    )

Nemathelminth