How to identify compiler before defining cython extensions?

Question

I'm building a platform independent cython project where I want to pass compiler args based on the compiler being used. I can guess the compiler based on platform or assume it's the same compiler used for Python but it's not guaranteed to match. Normally I inject into the cmdclass arg to setuptool's setup method and wrap the install or build_ext commands to check internal state. But in this case I have to cythonize the extension modules before I reach the wrappers.

Is there any way to determine the compiler inside setup.py before cythonizing the extension modules?

Tantalum

can't you pass the compiler as argument to setup.py: python setup.py build --compiler=mingw32? – Forbid

you can also use cmake to compile cython code in cross-platform manner: github.com/thewtex/cython-cmake-example – Forbid

@denfromufa You can pass --compiler=mingw32, but other recipients of the repository won't necessarily know what to set the compiler argument to, or if it's a dependency of another repo. And pip install definitely won't create such an argument to setuptools. I could read the argument if i were only using python setup.py install myself -- that is true. – Tantalum

Also I hadn't seen cython-cmake before -- I'll look into that. I'd rather not introduce a complicated CMake pattern into the repository if I can avoid it and would like to instead just set the appropriate compiler flags in setuptools based on the compiler that's been picked (by user or setuptools or python). But it appears to allow for compiler specific arguments after cythonization is defined. – Tantalum

Answer 1

After posting on the cython forums and searching for related issues in distutils I found this post showing how to move the compiler arguments into the build_ext assignment. If I subsequently remove all compiler arguments from the extension class I can now lazy assign them inside the command class as I expected. I can also get install and egg_info command classes to call my new version of the build_ext as well.

from setuptools.command.build_ext import build_ext

BUILD_ARGS = defaultdict(lambda: ['-O3', '-g0'])
for compiler, args in [
        ('msvc', ['/EHsc', '/DHUNSPELL_STATIC']),
        ('gcc', ['-O3', '-g0'])]:
    BUILD_ARGS[compiler] = args
    
class build_ext_compiler_check(build_ext):
    def build_extensions(self):
        compiler = self.compiler.compiler_type
        args = BUILD_ARGS[compiler]
        for ext in self.extensions:
            ext.extra_compile_args = args
        build_ext.build_extensions(self)

...
setup(
    ...
    cmdclass={ 'build_ext': build_ext_compiler_check })

Tantalum

The same thing seems to work with from setuptools.command.build_ext import build_ext (previous link had distutils which people consider deprecated or fully integrated into setuptools nowadays if I'm not mistaken; correct me if I'm wrong) – Lipase

That's correct. The setuptool import works exactly the same and should be used in place of the distutils import; though I don't know when distutils will actually disappear or is supposed to sunset as I've only seen it mentioned as a desire on python boards and not a requirement. – Tantalum

Answer 2

A simple variation of the first answer:

from setuptools import setup, Extension
from distutils.command.build_ext import build_ext

myextension = Extension(
    name = 'packagename',
    sources = [
        'source/debugger.cpp',
    ],
    include_dirs = [ 'source' ],
)

class build_ext_compiler_check(build_ext):
    def build_extensions(self):
        compiler = self.compiler.compiler_type

        # print('\n\ncompiler', compiler)
        if not 'msvc' in compiler:

            for extension in self.extensions:

                if extension == myextension:
                    extension.extra_compile_args.append( '-O0' )
                    extension.extra_compile_args.append( '-std=c++11' )

        super().build_extensions()

setup(
        name = 'packagename',
        version = __version__,
        ext_modules= [ myextension ],
    )

Bairam

Answer 3

(Sorry can't comment due to missing credit)

Unfortunately, the answer https://mcmap.net/q/1103732/-how-to-identify-compiler-before-defining-cython-extensions is slightly misleading as a build_ext instance's self.compiler.compiler_type is 'unix', not 'gcc' (the "distutils compiler_class").

I.e. the lookup from this defaultdict dictionary

BUILD_ARGS = defaultdict(lambda: ['-O3', '-g0'])
for compiler, args in [
    ('msvc', ['/EHsc', '/DHUNSPELL_STATIC']),
    ('gcc', ['-O3', '-g0'])]:
BUILD_ARGS[compiler] = args

wouldn't normally reach the 'gcc' entry but instead always fall back to the defaultdict's default (the lambda function).

That said you probably wouldn't notice this in a majority of cases as long as the default options remain the same as the 'gcc' options. E.g. clang seems to understand the same options as gcc.

It looks like you might be able to get at the actual invoked compiler name through self.compiler.compiler[0], though I haven't checked if this is reliable or portable.

Engineer