python c-api: create bytes using existing buffer without copying

Question

It seems to me the buffer protocol is more for exposing Python buffer to C.

I couldn't find a way to create a bytes object using existing buffer without copying in C.

Basically what I want is to implement something similar to PyBytes_FromStringAndSize() but without copying, and with a callback to free the buffer when the object is released. I don't know how big the buffer is before I receive the buffer returned from a C API. So creating bytes object in Python first and later fill it in is not an option.

I also looked into memoryview, PyMemoryView_FromMemory() doesn't copy but there is no way to pass a callback to free my buffer. And I'm not suse Python lib (e.g. Psycopg) can use memoryview object or not.

Do I have to create my own object to achieve these 2 requirements? Any other shortcut?

If I have to, how can I make sure this object works same as bytes so I can pass it to Python lib safely?

Thanks.

Pa

Answer 1

The only way to do this is to create a new object with PyBufferProcs* PyTypeObject.tp_as_buffer. I checked cpython source code thoroughly, as of 3.4.1, there is no out-of-box (so to speak) solution.

Pa

Could you please explain how one would do this, preferably with a code sample? – Newhall