Cross-compile extension on Linux for Windows

Question

I've managed to build some DLLs on Linux that are necessary for my Python extension using MinGW. Something along these lines:

from setuptools.command.build_py import build_py

class BuildGo(build_py):

    def run(self):
        if # need to build windows binaries
            self.build_win()
        build_py.run(self)

    def build_win(self):
        if # compilers and toolchain available
            try:
                # builds extra libraries necessary for this extension
            except subprocess.CalledProcessError as e:
                print(e.stderr)
                raise
            try:
                result = subprocess.check_output([
                    'x86_64-w64-mingw32-gcc-win32',
                    '-shared',
                    '-pthread',
                    '-o',
                    EXTRA_DLL,
                    FAKE_WIN_BINDINGS,
                    ARCHIVE_GENERATED_IN_PREVIOUS_STEP,
                    '-lwinmm',
                    '-lntdll',
                    '-lws2_32',
                ])
                print(result)
            except subprocess.CalledProcessError as e:
                print(e.stderr)
                raise

I was now hoping I could avoid extending build_ext in the same painful way to get it to cross-compile Cython code for Windows... I looked into the abyss of "elegant interplay of setuptools, distutils and cython", and before the abyss has a chance to look back into me... Isn't there a way to just specify some flag... like a name of compiler and Python binary for desired platform and... it would just do it?

I've read this article: http://whatschrisdoing.com/blog/2009/10/16/cross-compiling-python-extensions/ - it's almost 10 years old. And it just made me want to cry... did anything change since it was written? Or are these steps more or less what I'll have to do to compile for the platform other than the one I'm running on?

Or, is there an example project on the web which does it?

Goal

My ultimate goal is to produce an egg package which will contain both PE and ELF binaries in it and will install them in the correct location on either platform when installed by pip or pipenv. It should compile on Linux (compiling it on MS Windows isn't necessary).

Bonaparte

One problem is that Python on Windows is heavily tied to specific versions of MSVC and modules compiled with mingw generally aren't compatible. (There's been some effort recently to make a version of mingw that does create compatible Python modules, but it isn't perfect. You're in for a whole world of fun trying to use gfortran with Python on Windows, for example) – Prenotion

@Prenotion for a moment I thought to disregard your advise... I found this post: #32361619 and hoped I could do the same... but these tools don't work together any more, and the import library generated from Python DLL doesn't meet the mingw requirements... time for sackcloth and ashes. – Bonaparte

This is not possible to be done reliably that will not break in horrible, not easy to troubleshoot, ways. Compile the extension on Windows with the same MSVC as was used for that Python version. Mixing shared objects/modules built with different compilers is generally a big no-no. Even with only GCC the same version needs to be used, let alone between MingW and MSVC. – Mercaptide

@danny, @Prenotion do you happen to know why the infrastructure around this issue is so bad? I mean, Ruby is no different from Python in how it is set up in principle, but in Ruby world it's not a problem. Is this issue known to CPython developers? Had it been mentioned on the mailing list? I mean, this is so obviously broken and what we get in Python 3.7 is some renaming of type function? How does this make any sense... – Bonaparte

I know nothing about Ruby but this implies that the same applies with Ruby. I think the difference might be that they picked Mingw as the default option, which makes cross-compiling on Linux pretty straight forward. The Python devs would probably argue that MSVC is the "native" compiler and is freely available, so is a sensible choice. – Prenotion

Well, Ruby has DevKit, it takes care of giving you the correct compiler, compiling your extensions for users, it also can cross-compile your code from Linux to Windows and from Windows to Linux. None of that is true about Python. It's not really the same. Python alleges to be a free as in freedom language, but now it appears that if I want to write an extension that works for Windows users I must use non-free software... – Bonaparte

Yes, this is not anything specific to python. Like I mentioned, even linking objects built by a different version of GCC is not likely to work reliably. As different major versions of GCC have a different ABI, this makes sense. The same applies to different compilers. If you can target one particular cross platform toolchain, like with conda, these issues are non-existent and do not require any special code. I'd recommend conda for the above use case. – Mercaptide

@Mercaptide Sorry, you are confused. The problem isn't specific to Python. Its "solution" (or rather lack of one) is specific to Python. Ruby chose to solve this problem by using a compiler with API consistent across different platforms, and because of that cross-compilation is possible in Ruby. Python chose to use two inconsistent compiler API, and because of that cross-compilation is impossible. – Bonaparte

Yes, python chose to use MSVC on windows, which is the native compiler. That makes cross compilation on that platform with any other compiler impossible. This statement applies to everything compiled by different compilers on any platform and is not specific to python extensions. That is what is meant by not specific to python, ie the problem of linking modules built with different compilers. – Mercaptide

Answer 1

I'm posting this as community wiki because it's a pretty unsatisfactory answer: it only tells you why it's very hard rather than offers really solutions.

The official Python distributions on Windows are compiled with Microsoft Visual C (MSVC), and when compiling a Python extension it's generally necessary to use the same version as the one that Python was compiled with. This shows you that an exact compiler match is pretty important.

It is possible to get versions of Python compiled with Mingw, and these would then be compatible with modules compiled with Mingw. This could probably be made to work as a cross-compiler on Linux but the modules would only be useful to a very small subset of people that have this custom build of Python (so doesn't help create a useful distributable .egg file).

A reasonable effort has gone also into making a version of Mingw that can build compatible Python extensions on Windows: https://mingwpy.github.io/ (and I think also https://anaconda.org/msys2/m2w64-toolchain). The main driver for this seems to be the lack of freely Fortran compiler for Windows that is compatible with MSVC, hence the ability to build Fortran modules is very useful. The mingwpy toolchain worked pretty well in my experience, until Python 3.4 when the switch to a more recent version of MSVC brought a whole exciting new set of compatibility issues.

My feeling would be that any viable solution would probably be based around these mostly-working Mingw compilers for windows.

Prenotion

Just to add that conda was specifically designed with this problem in mind. It offers a cross platform toolchain that can build packages for Linux, OSX and Windows and greatly simplifies building and shipping cross platform native code extensions and other native code binaries and libraries. Its con is that it is not a 'native' feature of any OS so users will have to install it first and that it offers similar capabilities as pip binary wheels. That said, its bootstrap is simple and painless. – Mercaptide

You're welcome to edit that in to this answer (or post it separately) if you want. It doesn't quite solve the problem, but it's certainly a relevant alternative. – Prenotion

Answer 2

According to https://docs.python.org/3/distutils/builtdist.html , distutils only supports cross-compiling between win32 and win_amd64 as of this writing (3.7).

Moreover, building extensions with compilers other than the MSVC that Python is built with is not officially supported.

It is theoretically possible by getting a Linux toolchain for win32/64 (including the necessary headers and link libraries), a set of the necessary Python for Windows binaries to link against, then forge compiler and linker paths and/or options in setup.py -- though it will still be an unsupported setup.

So you'll be better off using a Windows VM or an online build service like AppVeyor.

Sewel

I've given up this idea long time ago. I didn't update my question, but I actually wrote to the Python mailing list only to discover that this is, basically, impossible. For now, I've switched to Anaconda, because it has... well, better toolchain than python.org. But, in the future, I'll try to stay away from Python when possible. – Bonaparte

Answer 3

Here is a proof of concept for cross compiling (Cython-) extensions for Windows on Linux, which follows more or less the steps for building with mingw-w64 on Windows.

But first a word of warning: While possible, the workflow is not really supported (it starts with the fact that the only supported windows compiler is MSVC), so it can be broken with changes in future versions. I use Python 3.7 for 64bit, things might be (slightly) different for other versions.

There might be legit scenarios for cross-compilation for Windows, but the python world seems to live quite good without, so probably cross-compilation is not the right direction in the most cases.

Preliminaries:

• Compiler: As at time of writing, the only real alternative is MinGW-w64 (e.g. sudo apt-get install mingw-w64) - the compiler for 64bit is x86_64-w64-mingw32-gcc.
• Headers: Python-headers on Linux and Windows are different (e.g. pyconfig.h), that means for cross-compiling the Python-headers from Windows are needed. The easiest way is to copy them from Windows-version for which the extension should be built.
• DLL: Handling of dynamic libraries is different on Windows and Linux. While on Linux one doesn't need the shared python library (it would even be wrong to use it as the python-symbols are provided by the python-executable built with -Xlinker -export-dynamic`) for linking, it is needed for Windows executable. mingw-w64's linker works differently than MSVC's: python-dll and not python-lib is needed.

distutils does not support mingw-w64, so we will perform all steps manually.

1. C code generation

Let's take the following simple Cython-extension foo.pyx

print("It is me!")

which can be transformed to C-code via:

>>> cython -3 foo.pyx

which creates the foo.c-file.

2. Compilation

The compilation step is:

>>> x86_64-w64-mingw32-gcc -c foo.c -o foo.o -I <path_to_windows_includes> -DMS_WIN64  -O2 <other compile flags>

I guess one can be minimalistic and only use -O2 compile flag in most cases. It is however important to define MS_WIN64-macro (e.g. via -DMS_WIN64). In order to build for x64 on windows it must be set, but it works out of the box only for MSVC (defining _WIN64 could have slightly different outcomes):

#ifdef _WIN64
#define MS_WIN64
#endif

3. Linking

The linking command is:

>>> x86_64-w64-mingw32-gcc -shared foo.o -o foo.pyd -L <path_to_windows_dll> -lpython37

It is important, that the python-library (python37) should be the dll itself and not the lib (see this SO-post).

One probably should add the proper suffix to the resulting pyd-file, I use the old convention for simplicity here.

4. Running:

Copying pyd-file to windows and now:

import foo
# prints "It is me!"

Done!

---

Embeded Python:

In case the python should be embeded, i.e. C-code is generated via cython -3 --embed foo.pyx, the compilation steps stays as above.

The linker step becomes:

>>> x86_64-w64-mingw32-gcc foo.o -o foo.exe -L <path_to_windows_dll> -lpython37 -municode

There are two noticeable differences:

• -shared is no longer should be used, as the result is no longer a dynamic library (that is what *.pyd-file is after all) but an executable.
• -municode is needed, because for Windows, Cython defines int wmain(int argc, wchar_t **argv) instead of int main(int argc, char** argv). Without this option, an error message like In function 'main': /build/mingw-w64-_1w3Xm/mingw-w64-4.0.4/mingw-w64-crt/crt/crt0_c.c:18: undefined reference to 'WinMain' collect2: error: ld returned 1 exit status would appear (see this SO-post for more information).

Note: for the resulting executable to run, a whole python-distribution (and not only the dll) is needed (see also this SO-post).

Demirep

Answer 4

I had the same issue once, but I just used a virtual machine to compile my most painfuly microsoft dependant programs.

https://developer.microsoft.com/en-us/windows/downloads/virtual-machines

If you don't have access to a windows machine or your programs uses very specific machiney like a fortran compiler optimized or some POSIX dependant stuff or newest features from VS redistributable versions, you better give a try to a virtual machine based compilation system.

Recite