PEP 3118 warning when using ctypes array as numpy array

Question

I'm getting the following warning message when I try to use a ctypes array as a numpy array:

Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes, numpy
>>> TenByteBuffer = ctypes.c_ubyte * 10
>>> a = TenByteBuffer()
>>> b = numpy.ctypeslib.as_array(a)
C:\Python27\lib\site-packages\numpy\ctypeslib.py:402: RuntimeWarning: Item size
computed from the PEP 3118 buffer format string does not match the actual item s
ize.
  return array(obj, copy=False)
>>> b
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint8)

The code seems to be working, though. Is it a bad idea to ignore this warning?

Background: I'm calling a C DLL that produces data in real time. I need to pass the DLL a series of buffers to hold the data. While waiting for the next buffer to fill, I'd like to process the most recent buffer with numpy and save the result. I'm generating the buffers with code like above, and things seem to be working, but I don't want to sweep an important problem under the rug.

Michal

Interestingly, this doesn't seem to happen with python2.6, only python2.7... – Unthankful

Answer 1

It's a bug in Python. ctypes currently produces invalid PEP 3118 type codes, which Numpy notices: http://bugs.python.org/issue10746 http://bugs.python.org/issue10744

When such an inconsistency is present, Numpy skips using the PEP 3118 buffer interface, and falls back to the old (obsolete) buffer interface. This should work properly.

You can silence the warning using Python's warnings module. However, the warning may have a performance impact.

You can also try working around the issue by wrapping the ctypes object in buffer().

Epicarp

Excellent, thank you. Glad to know it's a known bug and not me doing something dumb. – Michal

Can anyone comment on any change in performance by wrapping the ctypes objects with buffer()? Does this method reliably work? How does it differ from making a new numpy array from the ctype array EG: np.array(my_ctype_array[:]).reshape([back, to, original]) other thank mangling the shape? I thought ctypeslib.as_array() was the recommended practice, but the warning message is really annoying and it will turn off non-tech users. Thx. – Polymerous

Wrapping the ctypes objects with buffer() did not work for me EG: new = numpy.ctypeslib.as_array(buffer(old)) returns TypeError: buffer() takes at least 1 argument (0 given). Can anyone add more clarity to the buffer() wrap option? Thanks! – Polymerous

Answer 2

There is a more convenient way of doing this, which avoids the warning altogether:

Instead of creating the data as a ctypes array first and then converting it to a NumPy array, just create it as a NumPy array right away, and then use numpy.ctypeslib.ndpointer as type specifier in your ctypes prototype. As an example, let's say you have a C function called f which takes a char* and a size_t as arguments:

void f(char* buf, size_t len);

Your ctypes prototype would be

from numpy.ctypeslib import ndpointer
some_dll = ctypes.CDLL(...)
some_dll.f.argtypes = [ndpointer(numpy.uint8, flags="C_CONTIGUOUS"),
                       ctypes.c_size_t]
some_dll.f.restype = None

and you can call this function as

a = numpy.zeros(10, numpy.uint8)
f(a, a.size)

Harmon

Very nice! I haven't used argtypes before, I'll try this out. – Michal