Definition of PyBufferProcs in Python 2.7 when class implements PEP 3118

Question

I am in the process of extending the classes in our library (which supports Python 2.7) to support PEP 3118, which has been back-ported to 2.7.

From the documentation, I need to initialize the tp_as_buffer field to point to a PyBufferProcs. From the documentation for 2.7, however, the description of this structure only contains entries for the old buffer protocol. From the sources, I gather that PyBufferProcs has some additional entries for the new protocol (bf_getbuffer and bf_releasebuffer).

The questions remain:

• Do I have to do something special to tell Python that these new entries are valid?

• Do I have to fill in the entries for the old protocol? (The documentation for 2.7 says, for example, that bf_getsegcount may not be null. But this entry shouldn't be used if I'm supporting PEP 3118.)

Envelopment

Look at bytearray_as_buffer in 2.7, with the additional fields for bf_getbuffer and bf_releasebuffer. Also, PyByteArray_Type.tp_flags sets Py_TPFLAGS_HAVE_NEWBUFFER. PyObject_GetBuffer uses the PyObject_CheckBuffer macro to check that this flag is set. – Kummerbund

Yes absolutely, I have looked at exactly those things – but that isn’t the problem we are asking about (unless I am missing something, those are the public, official Python API funcs – which won’t respond properly to our python C-extension objects without the proper (and needlessly cryptically documented) stuff set up in the related PyBufferProcs (as appropriately pointed at in the type struct def). Help us with getting all of that proper and correct! – Falmouth

The question's "gather" link points at Python 3's bytearray implementation, which doesn't use the Python 2 version of PyBufferProcs that has six fields, including the last two for the new buffer protocol. Using just the new protocol only requires those two fields. The Python 3 source also doesn't use the Py_TPFLAGS_HAVE_NEWBUFFER flag; that's just in Python 2. Setting this flag is the "something special" that tells PyObject_CheckBuffer that the new entries are valid. – Kummerbund

@eryksun IS CORRECT. I got my objects Py_buffered this very morning with this. – Falmouth

Answer 1

You can just fill the last two fields of PyBufferProcs but you have to add the Py_TPFLAGS_HAVE_NEWBUFFER flag to the tp_flags of your types. This is the special thing that was introduced in python2 to make the new protocol available together with the old one.

I have no idea why this isn't documented anywhere, but you can see it used in the definition of the bytearray type for python 2.7 (see here):

    &bytearray_as_buffer,               /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
    Py_TPFLAGS_HAVE_NEWBUFFER,          /* tp_flags */

---

This content was already posted in comments, but it deserves an answer.

Obscure

It cannot be overstated how correct this is – the lack of documentation for Py_TPFLAGS_HAVE_NEWBUFFER is actually listed as a Python bug (q.v. bugs.python.org/issue23850) and, as @Obscure says, it’s the “special thing” one needs to unleash PEP 3118. Yes! – Falmouth