Change Cython's naming rules for .so files

Question

I'm using Cython to generate a shared object out of Python module. The compilation output is written to build/lib.linux-x86_64-3.5/<Package>/<module>.cpython-35m-x86_64-linux-gnu.so. Is there any option to change the naming rule? I want the file to be named <module>.so without the interpreter version or arch appendix.

Balls

should be same as setuptools build options, you can find it in python setup.py --help – Priedieu

@YOU: Thanks, I managed to override build/lib/temp directories, but I don't see anything related to platform suffix etc. – Balls

python setup.py build --build-platlib . – Priedieu

seems like it's ignored... – Balls

I know this is late but the suffix is there for good reasons. I don't see any good reason to remove it, and I'd recommend that people leave it rather than follow the advice in these answers. It's maybe useful to have the information out there on how to change the suffix, but no-one should think it's a good idea. – Entomo

@Entomo the background for this question was that distutils doesn't support PEP 384, which I've omit from the question. – Balls

Answer 1

Seems like setuptools provides no option to change or get rid of the suffix completely. The magic happens in distutils/command/build_ext.py:

def get_ext_filename(self, ext_name):
    from distutils.sysconfig import get_config_var
    ext_path = ext_name.split('.')
    ext_suffix = get_config_var('EXT_SUFFIX')
    return os.path.join(*ext_path) + ext_suffix

Seems like I will need to add a post-build renaming action.

---

Update from 08/12/2016:

Ok, I forgot to actually post the solution. Actually, I implemented a renaming action by overloading the built-in install_lib command. Here's the logic:

from distutils.command.install_lib import install_lib as _install_lib

def batch_rename(src, dst, src_dir_fd=None, dst_dir_fd=None):
    '''Same as os.rename, but returns the renaming result.'''
    os.rename(src, dst,
              src_dir_fd=src_dir_fd,
              dst_dir_fd=dst_dir_fd)
    return dst

class _CommandInstallCythonized(_install_lib):
    def __init__(self, *args, **kwargs):
        _install_lib.__init__(self, *args, **kwargs)

    def install(self):
        # let the distutils' install_lib do the hard work
        outfiles = _install_lib.install(self)
        # batch rename the outfiles:
        # for each file, match string between
        # second last and last dot and trim it
        matcher = re.compile('\.([^.]+)\.so$')
        return [batch_rename(file, re.sub(matcher, '.so', file))
                for file in outfiles]

Now all you have to do is to overload the command in the setup function:

setup(
    ...
    cmdclass={
        'install_lib': _CommandInstallCythonized,
    },
    ...
)

Still, I'm not happy with overloading standard commands; if you find a better solution, post it and I will accept your answer.

Balls

Answer 2

This behavior has been defined in distutils package. distutils uses sysconfig and "EXT_SUFFIX" config variable:

# Lib\distutils\command\build_ext.py

def get_ext_filename(self, ext_name):
    r"""Convert the name of an extension (eg. "foo.bar") into the name
    of the file from which it will be loaded (eg. "foo/bar.so", or
    "foo\bar.pyd").
    """
    from distutils.sysconfig import get_config_var
    ext_path = ext_name.split('.')
    ext_suffix = get_config_var('EXT_SUFFIX')
    return os.path.join(*ext_path) + ext_suffix

Starting with Python 3.5 "EXT_SUFFIX" variable contains platform information, for example ".cp35-win_amd64".

I have written the following function:

def get_ext_filename_without_platform_suffix(filename):
    name, ext = os.path.splitext(filename)
    ext_suffix = sysconfig.get_config_var('EXT_SUFFIX')

    if ext_suffix == ext:
        return filename

    ext_suffix = ext_suffix.replace(ext, '')
    idx = name.find(ext_suffix)

    if idx == -1:
        return filename
    else:
        return name[:idx] + ext

And custom build_ext command:

from Cython.Distutils import build_ext

class BuildExtWithoutPlatformSuffix(build_ext):
    def get_ext_filename(self, ext_name):
        filename = super().get_ext_filename(ext_name)
        return get_ext_filename_without_platform_suffix(filename)

Usage:

setup(
    ...
    cmdclass={'build_ext': BuildExtWithoutPlatformSuffix},
    ...
)

Youmans

Answer 3

the solution is very simple, just change one line in build_ext.py at

    def get_ext_filename(self, ext_name):

        from distutils.sysconfig import get_config_var
        ext_path = ext_name.split('.')
        ext_suffix = get_config_var('EXT_SUFFIX') 
        return os.path.join(*ext_path) + ext_suffix

change ext_suffix = get_config_var('EXT_SUFFIX') to ext_suffix = ".so" or to ".pyd" on Windows

that's it, you don't have to worry about the output name ever again

Alfons

I tried this, like this: setup(ext_modules=cythonize("my_function.pyx"), include_dirs=[numpy.get_include()], ext_suffix=".pyd"). Did not work. – Corelation