pyproject.toml and cython extension module

Question

I have an existing python project that is using mostly setup.py to build the project. The project has 2 x Cython extension modules configured in setup.py.

Initially I did pip install -e . for development, but since then I'm using python setup.py build_ext --inplace to rebuild only the extensions when needed. Which is much faster compared to installing the package.

I started migrating the project to pyproject.toml including the project configs in the [project] section in pyproject.toml

My setup.py basically only contains the Cython extension modules, which I understand can not be migrated to 'pyproject.toml' as of yet.

Now my problem: python setup.py build_ext --inplace doesn't work anymore, because setup.py doesn't have all the information, and is not consulting pyproject.toml to read the project config (hence project configs information are missing).

Do I need to revert to my original setup.py/*.cfg config, or is there some way to tell setup.py to retrieve the config from pyproject.toml?

Dupin

I don't know if one can use exclusively pyproject.toml to configure a package, but I've been using it together with setup.cfg, and setup.py, and am able to compile Cython extensions with python3 setup.py build_ext -i. Do you want me to add an answer with an example? – Vociferant

No thanks. That is what I'm doing as well. – Dupin

Answer 1

For me it is working doing how they suggest in the Cython documentation and in the setuptools documentation.

Adding cython as dependency in the requires list, was the only change I made to pyproject.toml.

Following is the content of setup.py:

from setuptools import setup, Extension
from Cython.Build import cythonize
from Cython.Compiler import Options
import numpy

# These are optional
Options.docstrings = True
Options.annotate = False

# Modules to be compiled and include_dirs when necessary
extensions = [
    # Extension(
    #     "pyctmctree.inpyranoid_c",
    #     ["src/pyctmctree/inpyranoid_c.pyx"],
    # ),
    Extension(
        "pyctmctree.domortho",
        ["src/pyctmctree/domortho.pyx"], include_dirs=[numpy.get_include()],
    ),
]


# This is the function that is executed
setup(
    name='mypackage',  # Required

    # A list of compiler Directives is available at
    # https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#compiler-directives

    # external to be compiled
    ext_modules = cythonize(extensions, compiler_directives={"language_level": 3, "profile": False}),
)

Note: get_include is needed only if you use the c version of numpy.

Once the setup.py has been created I can compile the Cython extensions using
pip install -e . (while inside the project directory).

So far I have noticed the following 2 draw-backs in using pip install -e .:

• The required package will be checked everytime
• Every .pyx file is built regardless of the timestamp

The above considerably slowdown development.

Faster alternatives are:

• python3 setup.py build_ext -i
• python3 setup.py develop (although deprecated)

Codification

Answer 2

Below is a little bit hack example to build the ext with pyproject.toml

pyproject.toml

[tool.setuptools]
py-modules = ["_custom_build"]

[tool.setuptools.cmdclass]
build_py = "_custom_build.build_py"

_custom_build.py

from setuptools import Extension
from setuptools.command.build_py import build_py as _build_py

class build_py(_build_py):
    def run(self):
        self.run_command("build_ext")
        return super().run()

    def initialize_options(self):
        super().initialize_options()
        if self.distribution.ext_modules == None:
            self.distribution.ext_modules = []

        self.distribution.ext_modules.append(
            Extension(
                "termial_random.random",
                sources=["termial_random/random.c"],
                extra_compile_args=["-std=c17", "-lm"],
            )
        )

Lalonde

This works well, with the exception that it results in builds without platform specification (e.g. PACKAGE-VERSION-py3-none-any.whl) instead of the expected PACKAGE-VERSION-cp311-cp311-win_amd64.whl). Any ideas? – Monkey

I think it's because build_py is the wrong cmdclass to subclass if you want a native extension. – Cuboid