Python objects as userdata in ctypes callback functions

Question

The C function myfunc operates on a larger chunk of data. The results are returned in chunks to a callback function:

int myfunc(const char *data, int (*callback)(char *result, void *userdata), void *userdata);

Using ctypes, it's no big deal to call myfunc from Python code, and to have the results being returned to a Python callback function. This callback work fine.

myfunc = mylib.myfunc
myfunc.restype = c_int
myfuncFUNCTYPE = CFUNCTYPE(STRING, c_void_p)
myfunc.argtypes = [POINTER(c_char), callbackFUNCTYPE, c_void_p]

def mycb(result, userdata):
    print result
    return True

input="A large chunk of data."
myfunc(input, myfuncFUNCTYPE(mycb), 0)

But, is there any way to give a Python object (say a list) as userdata to the callback function? In order to store away the result chunks, I'd like to do e.g.:

def mycb(result, userdata):
    userdata.append(result)

userdata=[]

But I have no idea how to cast the Python list to a c_void_p, so that it can be used in the call to myfunc.

My current workaround is to implement a linked list as a ctypes structure, which is quite cumbersome.

Pecos

Answer 1

I guess you could use the Python C API to do that... maybe you could use a PyObject pointer.

edit: As the op pointed out in the comments, there's already a py_object type readily available in ctypes, so the solution is to create first a ctypes.py_object object from the python list and then casting it to c_void_p to pass it as an argument to the C function (I think this step might be unnecessary as a parameter typed as void* should accept any pointer, and it would be faster to pass just a byref). In the callback, the reverse steps are done (casting from the void pointer to a pointer to py_object and then getting the value of the contents).

A workaround could be to use a closure for your callback function so it already knows in which list it has to append the items...

myfunc = mylib.myfunc
myfunc.restype = c_int
myfuncFUNCTYPE = CFUNCTYPE(STRING)
myfunc.argtypes = [POINTER(c_char), callbackFUNCTYPE]


def mycb(result, userdata):
    userdata.append(result)

input="A large chunk of data."
userdata = []
myfunc(input, myfuncFUNCTYPE(lambda x: mycb(x, userdata)))

Emelinaemeline

Well, the nice thing about ctypes is that everything can be done in plain Python. So the Python C API seems to be a step back. Or am I missing something? – Pecos

I meant the option could be creating a ctypes proxy of a C-API PyObject struct, that should be the internal representation of any Python object and passing it around... – Emelinaemeline

Strange. Your workaround with a closure works fine on Linux, but the same program crashes on Windows. – Pecos

Have you noticed he change in the signature of the callback function and changed that accordingly in your C files? – Emelinaemeline

Nope. That's certainly the problem. I wondered why it worked without the change on Linux. – Pecos

But I found the solution I was looking for: cUserdata=ctypes.cast(pointer(ctypes.py_object(pyUserdata)), c_void_p) and in the callback: pyUserdata=ctypes.cast(cUserdata,POINTER(ctypes.py_object)).contents.value. I guess that's what had in mind with the Python C API? – Pecos

About the first question, it probably has to do with the calling conventions, by examining what happened I'd say the parameter in Linux was expected in a register and in Windows in the stack (and as it was missing, the stack was corrupted at some point). Is your Linux 64 bit and your Windows 32 bit? – Emelinaemeline

About the second, that's more or less what I was suggesting, using the C internal representation of the Python object... Luckily enough, ctypes already had it implemented and you didn't have to write your own wrapper and some C code to get the actual pointer. – Emelinaemeline

Both 32 bit. But in fact, the Windows library uses WINFUNCTYPE, while the Linux version uses CFUNCTYPE. – Pecos

Would you mind to update your answer with the py_object? Again, thanks for the hint! – Pecos