Python C Extensions - Why must callable C functions take arguments and return PyObject *

Question

I'm just starting to play with Python C extensions and am curious as to why a C function, which is callable from Python must take 2 PyObject* arguments and return a PyObject*. I wrote the following "Hello World" extension:

#include <Python.h>

static PyObject *
hello_world(PyObject *self, PyObject *noargs)
{
   printf("Hello World\n");
   return Py_BuildValue("");
}


// Module functions table.

static PyMethodDef
module_functions[] = {
    { "hello_world", hello_world, METH_NOARGS, "hello world method" },
    { NULL }
};


// This function is called to initialize the module.
PyMODINIT_FUNC
inittesty2(void)
{
    Py_InitModule("testy2", module_functions);
}

Why can't I (especially with METH_NOARGS) use the following hello_world method:

static void
hello_world()
{
   printf("Hello World\n");
}

?

Aalesund

Answer 1

There are several things to say about the various PyObject pointers.

1. The one required as return type is used for the exception handling mechanism. Specifically, if your function returns a null pointer, the Python interpreter will throw an exception. (You should only do that after calling one of the PyErr_.. functions to set a specific exception.)

   This also means that whenever you do not want to throw an exception, you must return a pointer to some real PyObject. If there is nothing in particular your function is supposed to return, simply return Py_None (best use the Py_RETURN_NONE macro to get the reference count right), or "true" (using Py_RETURN_TRUE).

2. The first argument, PyObject *self points to the object the function is called from, or to the module instance it belongs to. Note that every function you define is either a class method, or a module method. There are no totally independent functions.

3. The second argument, PyObject *args points to the function argument (which may be a tuple or list of multiple arguments). You are right in pointing out that a function that does not take any arguments should not need this — and, as far as I can tell, you are right. You do not have to define it; you can simply define a function as

   static PyObject *PyMyClass_MyFunc(PyObject *self) {
     /* ..do something.. */
     Py_RETURN_TRUE;
   }
   

   You will still have to cast this to PyCFunction when you put it into the PyMethodDef for the data type you define, but I believe that cast is safe as long as you use the METH_NOARGS flag. But note the comments below for possible risks.

4. Finally, a function may in fact have a third argument like this:

   static PyObject *PyMyClass_Func(PyObject *self, PyObject *args, PyObject *kwds)
   {
     /*...*/
   }
   

   The third argument is used for named, optional arguments. In this case, too, you must cast the function pointer to PyCFunction, but that, too, is safe if you set the right flag (METH_KEYWORDS) for your function in the method table.

Subjunction

It's Py_None, not PyNone. Also there's Py_RETURN_NONE similar to Py_RETURN_TRUE and Py_RETURN_FALSE. – Reeba

For (3), correct me if I'm wrong but if the code was using a calling convention where all args are passed on the stack and the callee (the function) pops them off the stack, removing the unused argument would cause a crash because Python will always pass NULL as the second arg and the function would not pop it. – Reeba

Yes, that's because in cdecl which you're probably using, args are pushed right-to-left onto the stack and are poped by caller. So the extra NULL on the stack will be ignored by the function. That doesn't mean it will work on any other calling convention. For example it won't for stdcall. As long as portability is not an issue for OP, I guess this is OK. If it is, I wouldn't skip the args. – Reeba

@Reeba Ok, thanks for pointing this out. There might indeed be a risk I wasn't aware of. I have edited the post; it takes your comments into account now. – Subjunction

For what it's worth, (3) worked when I tried it, but the compiler did give me the following warning - " warning: initialization from incompatible pointer type [enabled by default]" when I tried to declare the 'arg-less' function in the module functions array – Aalesund

Answer 2

The first argument for module level functions is the module object. When defining classes in C (the same PyMethodDef structure is used for methods there), the first argument is the instance (similar to self in Python).

When using METH_NOARGS, Python will pass NULL as the second argument. They could cast it to a function with one argument but I guess they didn't think it's needed.

The return value is easy to explain. Every Python function has a return value. If you don't explicitly use return in Python, the function will return None.

Of course in C you have to be explicit about the return value so if you don't use it, you have to return None yourself. Python provides a macro for this:

Py_RETURN_NONE;

Alternatively you could access the global None instance yourself:

Py_INCREF(Py_None);
return Py_None;

but the macro is easier to use.

You could think that returning NULL should be equivalent to None but NULL is used to indicate that the function raised an exception.

Reeba

Answer 3

Because Python functions, even trivial ones that just print to stdout, are more than just wrappers around C functions.

In the simplest case, think about the introspection facilities in Python. A Python function is a full-fledge object that you can query:

>>> def hello():
...     print 'hello'
... 
>>> dir(hello)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']

You could of course imagine an extension facility that just wrapped C functions. Check out SWIG, which lets your write extensions for lots of scripting languages, including Python. Because it's going for the lowest common denominator, it'll let you wrap a function like your hello_world, but of course you lose a lot of power, too.

Shep