Using Py_buffer and PyMemoryView_FromBuffer with different itemsizes

This question is related to a previous question I asked. Namely this one if anyone is interested. Basically, what I want to do is to expose a C array to Python using a Py_buffer wrapped in a memoryview-object. I've gotten it to work using PyBuffer_FillInfo (work = I can manipulate the data in Python and write it to stdout in C), but if I try to roll my own buffer I get a segfault after the C function returns.

I need to create my own buffer because PyBuffer_FillInfo assumes that the format is char, making the itemsize field 1. I need to be able to provide items of size 1, 2, 4 and 8.

Some code, this is a working example:

Py_buffer *buf = (Py_buffer *) malloc(sizeof(*buf));
int r = PyBuffer_FillInfo(buf, NULL, malloc(sizeof(char) * 4), 4, 0, PyBUF_CONTIG);
PyObject *mv = PyMemoryView_FromBuffer(buf);
//Pack the memoryview object into an argument list and call the Python function
for (blah)
  printf("%c\n", *buf->buf++); //this prints the values i set in the Python function

Looking at the implementation of PyBuffer_FillInfo, which is really simple, I rolled my own function to be able to provide custom itemsizes:

//buffer creation function
Py_buffer *getReadWriteBuffer(int nitems, int itemsize, char *fmt) {
  Py_buffer *buf = (Py_buffer *) malloc(sizeof(*buf));
  buf->obj = NULL
  buf->buf = malloc(nitems * itemsize);
  buf->len = nitems * itemsize;
  buf->readonly = 0;
  buf->itemsize = itemsize;
  buf->format = fmt;
  buf->ndim = 1;
  buf->shape = NULL;
  buf->strides = NULL;
  buf->suboffsets = NULL;
  buf->internal = NULL;
  return buf;
}

How i use it:

Py_buffer *buf = getReadWriteBuffer(32, 2, "h");
PyObject *mv = PyMemoryView_FromBuffer(buf);
// pack the memoryview into an argument list and call the Python function as before

for (blah)
  printf("%d\n", *buf->buf); //this prints all zeroes even though i modify the array in Python

return 0;
//the segfault happens somewhere after here

The result of using my own buffer object is a segfault after the C function returns. I really don't understand why this happens at all. Any help would be most appreciated.

EDIT According to this question, which I failed to find before, itemsize > 1 might not even be supported at all. Which makes this question even more interesting. Maybe I could use PyBuffer_FillInfo with a large enough block of memory to hold what I want (32 C floats for example). In that case, the question is more about how to assign Python floats to the memoryview object in the Python function. Questions questions.

So, in lack of answers I decided to take another approach than the one I originally intended. Leaving this here in case someone else hits the same snag.

Basically, instead of creating a buffer (or bytearray, equiv.) in C and passing it to Python for the extension user to modify. I simply redesigned the code slightly, so that the user returns a bytearray (or any type that supports the buffer interface) from the Python callback function. This way I need not even worry about the size of the items since, in my case, all the C code does with the returned object is to extract its buffer and copy it to another buffer with a simple memcpy.

Code:

PYGILSTATE_ACQUIRE; //a macro i made
PyObject *result = PyEval_CallObject(python_callback, NULL);
if (!PyObject_CheckBuffer(result))
  ; //raise exception

Py_buffer *view = (Py_buffer *) malloc(sizeof(*view));
int error = PyObject_GetBuffer(result, view, PyBUF_SIMPLE);
if (error)
  ; //raise exception

memcpy(my_other_buffer, view->buf, view->len);

PyBuffer_Release(view);
Py_DECREF(result);
PYGILSTATE_RELEASE; //another macro

I hope this helps someone.

Recommended topics

Hot tags