using a C extension in python, without installing it as a module

Question

I am writing C extensions for python. I am just experimenting for the time being and I have written a hello world extension that looks like this :

#include <Python2.7/Python.h>

static PyObject* helloworld(PyObject* self)
{
    return Py_BuildValue("s", "Hello, Python extensions!!");
}

static char helloworld_docs[] = "helloworld( ): Any message you want to put here!!\n";

static PyMethodDef helloworld_funcs[] = {
    {"helloworld", (PyCFunction)helloworld, METH_NOARGS, helloworld_docs},
    {NULL,NULL,0,NULL}
};

void inithelloworld(void)
{
    Py_InitModule3("helloworld", helloworld_funcs,"Extension module example!");
}

the code works perfectly fine, after installing it from a setup.py file I wrote, and installing it from command line

python setup.py install

What I want is the following :

I want to use the C file as a python extension module, without installing it, that is I want to use it as just another python file in my project, and not a file that I need to install before my python modules get to use its functionality. Is there some way of doing this ?

Berserk

Can you elaborate on exactly what you want to achieve? An extension module is simply a shared library that Python can load. "Installing" it simply means copying it to a location where Python looks for its modules. Do you mean that you want to run your module from a non-system-wide location? Or that you don't want to avoid relying on setup.py, and install (copy) the extension module manually instead? – Aundrea

It will need to exist on the platform somwhere, so can you clarify what you mean when you state you don't want to install it ? – Kioto

I want to use it as just another python file in my project, and not a file that I need to install before my other python modules in my project get to use its functionality – Berserk

@TonySuffolk66 can I do something like my python script will first check whether the module is installed, and if not then install the C module from the python script itself. Is there some way in which this can be achieved ? – Berserk

@Berserk - in theory you could do that - but why would you want to ? It is usual to install all dependencies with the application. I might be mistaken but I don't know of any major application which does on demand installation of dependencies (I am not talking about optional plugins). – Kioto

@TonySuffolk66 I am just testing. It is not a big application and I am not distributing it or anything. I'm just curious. So if I were to do it, how would I do it ? Thanks for your help so far by the way. – Berserk

Answer 1

You can simply compile the extension without installing (usually something like python setup.py build). Then you have to make sure the interpreter can find the compiled module (for example by copying it next to a script that imports it, or setting PYTHONPATH).

Apocopate

I don't think that is a solution, since you have to make sure python will find your module, which means, you have to install it at the end, or at least extend your path. – Vig

@Apocopate I built the extension using python setup.py build and it gave me shared libraries (.so files) for mac osx 10.6 and mac osx 10.10 in two different folders. If I wanted to use my compiled extension on windows, how would I do so ? and also once I built it, how do I use the shared library ? – Berserk

@Berserk You can't use the modules compiled for Mac on Windows. C extension modules have to be compiled for the target system and Python version that should use them. – Mealy

Answer 2

You can create your "own interpreter" by not extending python, but embedding it into your application. In that way, your objects will be always available for the users who are running your program. This is a pretty common thing to do in certain cases, for example look at the Blender project where all the bpy, bmesh and bge modules are already included.

The downside is, your users can't use the python command directly, they have to use your hello_world_python instead. (But of course you can provide your extension as a module as well.) And that also means, you have to compile and distribute your application for all platforms you want to support -- in case you want to distribute it as a binary, to make your users lives a bit easier.

For further information on embedding python into your program, read the propriate sections of the documentation:

Embedding Python in Another Application

Personal suggestion: Use Python 3.5 whenever you can, and stop supporting the old 2.x versions. For more information, read this article: Should I use Python 2 or Python 3 for my development activity?

Vig

why exactly is the -1? – Vig

can I do something like my python script will first check whether the module is installed, and if not then install the C module from the python script itself. Is there some way in which this can be achieved ? – Berserk

@Berserk you can try/except the import of your module and warn the user on ImportError, so that s/he must install your module. You can even place further instructions in the given error message (like where to download it from). – Vig

(another possible solution is -- which is also very common -- implement a pure python version of your module as a fallback option, so even if your C extension is not available, but your pure python version is, then your program will use that instead. Ofc if that one is missing yoo you have to do what I mentioned in the previous comments of mine) – Vig

in your mentioned try/except approach, could I install the module in some way in the except part, if I got an ImportError in the try part? Thanks for your help so far by the way. – Berserk