Making an executable in Cython

Question

Been playing with cython. Normally program in Python, but used C in a previous life. I can't figure out how to make a free-standing executable.

I've downloaded cython, and I can make a .pyx file (that's just a normal Python file with a .pyx extension), that executes in the Python shell, using: import pyximport; pyximport.install()

I can generate a .c file at the command line with: cython file.pyx I can generate a .so file by building a standard setup.py and executing:

setup.py build_ext --inplace

I've tried making an executable out of the .so file using gcc with various options, but always have tons of missing files, headers, etc. Have tried pointing to headers from virtually everywhere, but with no success, and am not really familiar with what all the gcc options do, or even if I should be using gcc.

I've having a disconnect here with the fact that I can run my program in the Python shell, but not at the command line, (I don't want users to have to get into the shell, import modules, etc).

What am I missing here?

Fibrinolysin

cython myfile.pyx (without options) generates a .c file. When I use the --embed option it generates a .c file that's a couple 100 lines larger. Diff'ing them shows that a bunch of ifdef's were added. I'm still left with the problem: How do I use this .so? I can't seem to get the right combination of compiler options. Reading through the cython documentation for numpy, one way is to make a wrapper python program that imports the .so. I'm surprised that when I hide the .c and .pyx file and then say: import NameOfMyFile with no extensions, it seems to select the .so file. – Fibrinolysin

Possible duplicate of Compile main Python program using Cython – Epiblast

@Epiblast Also linked to Can Cython compile to an EXE? – Chanda

Answer 1

What you want is the --embed flag for the Cython compiler. There isn't a ton of documentation on it, but this is what I was able to find. It does link to a simple working example.

To compile the Cython source code to a C file that can then be compiled to an executable you use a command like cython myfile.pyx --embed and then compile with whichever C compiler you are using.

When you compile the C source code, you will still need to include the directory with the Python headers and link to the corresponding Python shared library on your system (a file named something like libpython27.so or libpython27.a if you are using Python 2.7).

Edit: Here are some more instructions on how to get the commands for including the proper headers and linking against the proper libraries.

As I said earlier, you need to run the Cython compiler like this:

cython <cython_file> --embed

To compile using gcc, you will need to find where the python headers are on your system (you can get this location by running distutils.sysconfig.get_python_inc() (you'll have to import it first). It is probably just the /include subdirectory in your Python installation directory.

You will also have to find the python shared library. For Python 2.7 it would be libpython27.a on Windows or libpython2.7.so on Linux.

Your gcc command will then be

gcc <C_file_from_cython> -I<include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>

It may be wise to include the -fPIC flag. On Windows 64 bit machines you will also have to include the flags -D MS_WIN64 that tells mingw to compile for 64 bit windows.

If you are compiling something that depends on NumPy, you will also need to include the directory containing the NumPy headers. You can find this folder by running numpy.get_include() (again, after importing numpy). Your gcc command then becomes

gcc <C_file_from_cython> -I<include_directory> -I<numpy_include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>

This gcc command option guide may be helpful.

Also, I would recommend you use Cython memory views if possible. That will make it so that you won't have to include the NumPy headers and include the NumPy pxd file in your Cython file. It also makes slicing operations easier for the C compiler to optimize.

Silin

lanH, I appreciate all the suggestions, and I am getting close. My command is: gcc -I [[python2.7inclues] -o a.out helloworld.c -lpython2.4. I now only have 2 errors: Undefined references to Py_InitModule4_64 and PyUnicodeUCS2_DecodeUTF8. Google'ing suggests that this is a confusion of 32 and 64 bit. I'm on Linux RH5. – Fibrinolysin

Adding -fPIC didn't make any difference. – Fibrinolysin

Looks like you need to add -D MS_WIN64 as suggested in one of the answers to this question -fPIC is a flag to make the generated binaries a little friendlier for the operating system. If I remember right it avoids an error I ran into some time ago when building Python modules on Linux. Why are you using the Python 2.7 includes and the Python 2.4 library? That could be causing trouble as well. – Silin

The -D option didn't affect anything. I'm including Python2.4, cause I get "can't find" errors using 2.5,.6 and .7. I tried including the full path, but says it still can't find it. The Sys Admin. is doing a full update now--perhaps that will help. Again, thanks for the advice. – Fibrinolysin

What I don't understand is: When in the python shell, importing the .pyx file causes it to compile, load and execute. Why can't I do the same thing on the command line? – Fibrinolysin

I suppose if you are willing to depend on a Python file, you could just do that. You could just make a python script that uses pyximport to import the cython script. It seems like a bit of a hack, but it will probably work for most simple scripts. I think one of the advantages of doing it with gcc is that you can get a standalone executable that doesn't depend on Python, I don't have a python-free system to verify that on right now though. You may have to add static linking for that. Pyximport doesn't let you customize the setup script, so that approach would also be limited to small projects. – Silin

Hmm. Judging by this question something more is needed to make a full standalone executable for systems without Python installed. Perhaps compiling a Cython extension module, running it from Python, and then using pyinstaller might work. I don't know at this point though. – Silin

@HaydenDarcy I originally tested this with MinGW, though MinGW isn't compatible with recent Python releases and even for older versions libpython.a may need separate installation. A similar approach ought to work fine for MSVC. You'll have to change to MSVC style flags and link to libpython.dll instead of libpython.a or libpython.so. The main idea is that Cython can generate code for a standalone executable, but when you compile the generated file you have to include the Python headers and link against the Python runtime (and make sure both can be found in the paths searched by the compiler). – Silin

Don't forget to add Chrome driver changes as it won't work on another pc as well as other issues (as I found out). It can work though it's not as simple as I imagined. Alternatively, distribute Winpython barebones + python jobs. Add cython if you want security. As people will find out, trying to get python to run without python is understandably complex and bug prone – Wallah

Pyinstaller works quite well so long as you provide correct pathing for chrome driver. However the issues arise when combining pyinstaller + Cython. See: #47027149 . The file is unbelievably massive. Winpython + standard python files could also work. I am yet to see a working standalone cython or nuitka solution, though both appear to be buggy to get working by there selves – Wallah

Right, one way or another you end up having to rely on the interpreter and corresponding runtime. Redistributing those makes files large. Even static compilation approaches rely on the interpreter and focus on compiling extensions rather than doing some sort of source-to-source transformation. That kind of thing isn't possible for arbitrary user code because of all the dynamic features in Python. If a given executable can have arbitrary Python semantics embedded in it, it's already implementing some kind of interpreter. There's no way around that for arbitrary user code. – Silin

Does anyone have a complete reprodudible example in a repo with a makefile? – Maundy

github.com/cython/cython/tree/… – Silin

Answer 2

Tested this on Ubuntu:

Install Cython using the following (Python 2):

sudo apt-get install cython

For Python 3:

sudo apt-get install cython3

To compile the Python code into a C code run (change cython to cython3 for Python 3):

cython --embed -o example.c example.py

This will generate the example.c file.

Now compile the example.c file:

gcc -Os -I /usr/include/python2.7 example.c -lpython2.7 -o example

Run the file:

./example

Now for Python 3, something like this will work (not tested):

gcc -Os -I /usr/include/python3.6 example.c -lpython3.6 -o example

where python3.x is the version of Python installed on your machine.

Tapes

And this load all the imports and modules that where made with cython right? – Signorina

If you mean the Python imports and modules, yes. – Tapes

How can I generate standalone file from multiple .py files? I have multiple python files and I want to compile them all into single package using cython. – Knelt

Did you tried compiling the main file that imports all the others? – Tapes

And just make sure the .py-file you're going to embed does not contain a - in the name. Use a name like foo_bar.py (and not foo-bar.py). – Walhalla

I got it working on Mac thanks to this post. Here is what I did in case anyone needs a sample: github.com/sarnobat/cython/blob/main/run.sh – Recant

Answer 3

This is a solution for Windows + MS Visual Studio 14 (as no-one mentioned the cl.exe parameters yet).

First use the embed parameter to produce the test.c file:

cython test.pyx --embed

Then compile it:

call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
cl test.c /I C:\Python37\include /link C:\Python37\libs\python37.lib

The output is small executable file test.exe (140 KB in my case for a print("Hello World")).

Notes:

• for a GUI app (and no console app), see @longgangfan's answer Can Cython compile to an EXE?, i.e. add /subsystem:windows /entry:wmainCRTStartup in the cl.exe parameters.

• for more information about whether the source code is still there, see Are executables produced with Cython really free of the source code? and Does compiled standalone Cython executable still contain all original source code?.

• also, the .exe won't run out-of-the-box on any machine; some other files are required, see Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine.

Chanda

Answer 4

Building off of Rafael's answer, here's an explanation of how to make a 'Hello, World' executable with Cython and Python 3 (tested on Termux), followed by instructions for Python 2:

cython3 doens't seem to be an option on my system, so here's what I do instead:

The file is called test.py:

print("Hello, world!")

Then do this:

cython -3 --embed -o test.c test.py

Then you need to figure out what include path you need, and maybe your libs.

I'm using Python 3.10.6; so, I use a program called python3.10-config to get the information before inputting it in gcc:

python3.10-config --includes

For me, that outputs this:

-I/data/data/com.termux/files/usr/include/python3.10 -I/data/data/com.termux/files/usr/include/python3.10

It gave me the same path twice (I don't know why), so just use that path (you only need to use it once, I think).

Then do

python3.10-config --libs

For me, that outputs this:

-lpython3.10 -lcrypt -ldl  -lm -lm

I'm not sure that every last one of those is important for every program, since the 'Hello, world' one works if I only put -lpython3.10.

Anyway, then I do this and it gives me an executable:

gcc -Os -I/data/data/com.termux/files/usr/include/python3.10 test.c -lpython3.10 -o test

There's a program called python2.7-config, too.

If I want to get an executable with Python 2.7.18, I can do this instead of the above:

cython -2 --embed -o test.c test.py

Then

python2.7-config --includes
python2.7-config --libs

That gives me these:

• -I/data/data/com.termux/files/usr/include/python2.7 -I/data/data/com.termux/files/usr/include/python2.7
• -lpython2.7 -ldl -lm

Finally, I do this to get the executable:

gcc -Os -I/data/data/com.termux/files/usr/include/python2.7 test.c -lpython2.7 -o test

Lidalidah