Python setuptools multiple extension modules with shared C source code building in parallel

Question

I'm working on a Python project with a setup.py that has something like this¹:

setup(
    cmdclass={"build_ext": my_build_ext},
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

I'm running into a problem when building the modules in parallel where it seems like one module tries to read common.o/common.obj while another is compiling it, and it fails. Is there some way to get setuptools to compile the C files for each module into their own build directories so that they aren't overwriting each other?

 

1. The actual project is more complicated with more modules and source files.

Herzel

setuptools.pypa.io/en/latest/userguide/… There is no. You can try to do something with extra_compile_args. Or stop building in parallel. – Vesper

Answer 1

I found a potential solution by overriding build_extension() in a custom build_ext class:

import copy, os
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext
class my_build_ext(build_ext):
    def build_extension(self, ext):
        # Append the extension name to the temp build directory
        # so that each module builds to its own directory.
        # We need to make a (shallow) copy of 'self' here
        # so that we don't overwrite this value when running in parallel.
        self_copy = copy.copy(self)
        self_copy.build_temp = os.path.join(self.build_temp, ext.name)
        build_ext.build_extension(self_copy, ext)
setup(
    cmdclass={"build_ext": my_build_ext},
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

I've also since been told of the (currently undocumented) libraries parameter for setup():

from setuptools import Extension, setup
setup(
    libraries=[
        ("common", {"sources": ["common.c"]}),
    ],
    ext_modules=[
        Extension("A", sources=["a.c"], libraries=["common"]),
        Extension("B", sources=["b.c"], libraries=["common"]),
    ],
)

Both solutions worked for me, but in slightly different ways. The first solution recompiles the code for each module, which allows you to specify different parameters to use for each module (ex. different defs). The second solution only has to compile to code once and it will reuse that for every module.

Herzel

I think that is the exactly proper way to handle that since build root is not exposed. – Su