Extracting unsigned char from array of numpy.uint8

Question

I have code to extract a numeric value from a python sequence, and it works well in most cases, but not for a numpy array.

When I try to extract an unsigned char, I do the following

unsigned char val = boost::python::extract<unsigned char>(sequence[n]);

where sequence is any python sequence and n is the index. I get the following error:

TypeError: No registered converter was able to produce a C++ rvalue of type 
unsigned char from this Python object of type numpy.uint8

How can I successfully extract an unsigned char in C++? Do I have to write/register special converters for numpy types? I would rather use the same code that I use for other python sequences, and not have to write special code that uses the PyArrayObject*.

Candida

Numpy uses native c types, so your goal is not going to be to convert the value but to just use it directly (by figuring out what it's memory location is, for instance). – Continuation

sequence is a boost::python::object, should I use static_cast instead? Like unsigned char val = static_cast<unsigned char>(sequence[n]); – Candida

Answer 1

One can register a custom from-python converter with Boost.Python that handles conversions from NumPy array scalars, such as numpy.uint8, to C++ scalars, such as unsigned char. A custom from-python converter registration has three parts:

• A function that checks if a PyObject is convertible. A return of NULL indicates that the PyObject cannot use the registered converter.
• A construct function that constructs the C++ type from a PyObject. This function will only be called if converter(PyObject) does not return NULL.
• The C++ type that will be constructed.

Extracting the value from the NumPy array scalar requires a few NumPy C API calls:

• import_array() must be called within the initialization of an extension module that is going to use the NumPy C API. Depending on how the extension(s) are using the NumPy C API, other requirements for importing may need to occur.
• PyArray_CheckScalar() checks if a PyObject is a NumPy array scalar.
• PyArray_DescrFromScalar() gets the data-type-descriptor object for an array scalar. The data-type-descriptor object contains information about how to interpret the underlying bytes. For example, its type_num data member contains an enum value that corresponds to a C-type.
• PyArray_ScalarAsCtype() can be used to extract the C-type value from a NumPy array scalar.

---

Here is a complete example demonstrating using a helper class, enable_numpy_scalar_converter, to register specific NumPy array scalars to their corresponding C++ types.

#include <boost/cstdint.hpp>
#include <boost/python.hpp>
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <numpy/arrayobject.h>

// Mockup functions.

/// @brief Mockup function that will explicitly extract a uint8_t
///        from the Boost.Python object.
boost::uint8_t test_generic_uint8(boost::python::object object)
{
  return boost::python::extract<boost::uint8_t>(object)();
}

/// @brief Mockup function that uses automatic conversions for uint8_t.
boost::uint8_t test_specific_uint8(boost::uint8_t value) { return value; }

/// @brief Mokcup function that uses automatic conversions for int32_t.
boost::int32_t test_specific_int32(boost::int32_t value) { return value; }


/// @brief Converter type that enables automatic conversions between NumPy
///        scalars and C++ types.
template <typename T, NPY_TYPES NumPyScalarType>
struct enable_numpy_scalar_converter
{
  enable_numpy_scalar_converter()
  {
    // Required NumPy call in order to use the NumPy C API within another
    // extension module.
    import_array();

    boost::python::converter::registry::push_back(
      &convertible,
      &construct,
      boost::python::type_id<T>());
  }

  static void* convertible(PyObject* object)
  {
    // The object is convertible if all of the following are true:
    // - is a valid object.
    // - is a numpy array scalar.
    // - its descriptor type matches the type for this converter.
    return (
      object &&                                                    // Valid
      PyArray_CheckScalar(object) &&                               // Scalar
      PyArray_DescrFromScalar(object)->type_num == NumPyScalarType // Match
    )
      ? object // The Python object can be converted.
      : NULL;
  }

  static void construct(
    PyObject* object,
    boost::python::converter::rvalue_from_python_stage1_data* data)
  {
    // Obtain a handle to the memory block that the converter has allocated
    // for the C++ type.
    namespace python = boost::python;
    typedef python::converter::rvalue_from_python_storage<T> storage_type;
    void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

    // Extract the array scalar type directly into the storage.
    PyArray_ScalarAsCtype(object, storage);

    // Set convertible to indicate success. 
    data->convertible = storage;
  }
};

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Enable numpy scalar conversions.
  enable_numpy_scalar_converter<boost::uint8_t, NPY_UBYTE>();
  enable_numpy_scalar_converter<boost::int32_t, NPY_INT>();

  // Expose test functions.
  python::def("test_generic_uint8",  &test_generic_uint8);
  python::def("test_specific_uint8", &test_specific_uint8);
  python::def("test_specific_int32", &test_specific_int32);
}

Interactive usage:

>>> import numpy
>>> import example
>>> assert(42 == example.test_generic_uint8(42))
>>> assert(42 == example.test_generic_uint8(numpy.uint8(42)))
>>> assert(42 == example.test_specific_uint8(42))
>>> assert(42 == example.test_specific_uint8(numpy.uint8(42)))
>>> assert(42 == example.test_specific_int32(numpy.int32(42)))
>>> example.test_specific_int32(numpy.int8(42))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    example.test_specific_int32(numpy.int8)
did not match C++ signature:
    test_specific_int32(int)
>>> example.test_generic_uint8(numpy.int8(42))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No registered converter was able to produce a C++ rvalue of type
  unsigned char from this Python object of type numpy.int8

A few things to note from the interactive usage:

• Boost.Python was able to extract boost::uint8_t from both numpy.uint8 and int Python objects.
• The enable_numpy_scalar_converter does not support promotions. For instance, it should be safe for test_specific_int32() to accept a numpy.int8 object that is promoted to a larger scalar type, such as int. If one wishes to perform promotions:
  • convertible() will need to check for compatible NPY_TYPES
  • construct() should use PyArray_CastScalarToCtype() to cast the extracted array scalar value to the desired C++ type.

Remonstrate

Answer 2

Here's a slightly more generic version of the accepted answer:
https://github.com/stuarteberg/printnum

(The converter is copied from the VIGRA C++/Python bindings.)

The accepted answer points out that it does not support casting between scalar types. This converter will allow implicit conversion between any two scalar types (even, say, int32 to int8, or float32 to uint8). I think that's generally nicer, but there is a slight convenience/safety trade-off made here.

#include <iostream>
#include <boost/python.hpp>

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION

// http://docs.scipy.org/doc/numpy/reference/c-api.array.html#importing-the-api
#define PY_ARRAY_UNIQUE_SYMBOL printnum_cpp_module_PyArray_API
#include <numpy/arrayobject.h>
#include <numpy/arrayscalars.h>


/*
 * Boost python converter for numpy scalars, e.g. numpy.uint32(123).
 * Enables automatic conversion from numpy.intXX, floatXX
 * in python to C++ char, short, int, float, etc.
 * When casting from float to int (or wide int to narrow int),
 * normal C++ casting rules apply.
 *
 * Like all boost::python converters, this enables automatic conversion for function args
 * exposed via boost::python::def(), as well as values converted via boost::python::extract<>().
 *
 * Copied from the VIGRA C++ library source code (MIT license).
 * http://ukoethe.github.io/vigra
 * https://github.com/ukoethe/vigra
 */
template <typename ScalarType>
struct NumpyScalarConverter
{
    NumpyScalarConverter()
    {
        using namespace boost::python;
        converter::registry::push_back( &convertible, &construct, type_id<ScalarType>());
    }

    // Determine if obj_ptr is a supported numpy.number
    static void* convertible(PyObject* obj_ptr)
    {
        if (PyArray_IsScalar(obj_ptr, Float32) ||
            PyArray_IsScalar(obj_ptr, Float64) ||
            PyArray_IsScalar(obj_ptr, Int8)    ||
            PyArray_IsScalar(obj_ptr, Int16)   ||
            PyArray_IsScalar(obj_ptr, Int32)   ||
            PyArray_IsScalar(obj_ptr, Int64)   ||
            PyArray_IsScalar(obj_ptr, UInt8)   ||
            PyArray_IsScalar(obj_ptr, UInt16)  ||
            PyArray_IsScalar(obj_ptr, UInt32)  ||
            PyArray_IsScalar(obj_ptr, UInt64))
        {
            return obj_ptr;
        }
        return 0;
    }

    static void construct( PyObject* obj_ptr, boost::python::converter::rvalue_from_python_stage1_data* data)
    {
        using namespace boost::python;

        // Grab pointer to memory into which to construct the C++ scalar
        void* storage = ((converter::rvalue_from_python_storage<ScalarType>*) data)->storage.bytes;

        // in-place construct the new scalar value
        ScalarType * scalar = new (storage) ScalarType;

        if (PyArray_IsScalar(obj_ptr, Float32))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, Float32);
        else if (PyArray_IsScalar(obj_ptr, Float64))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, Float64);
        else if (PyArray_IsScalar(obj_ptr, Int8))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, Int8);
        else if (PyArray_IsScalar(obj_ptr, Int16))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, Int16);
        else if (PyArray_IsScalar(obj_ptr, Int32))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, Int32);
        else if (PyArray_IsScalar(obj_ptr, Int64))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, Int64);
        else if (PyArray_IsScalar(obj_ptr, UInt8))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, UInt8);
        else if (PyArray_IsScalar(obj_ptr, UInt16))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, UInt16);
        else if (PyArray_IsScalar(obj_ptr, UInt32))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, UInt32);
        else if (PyArray_IsScalar(obj_ptr, UInt64))
            (*scalar) = PyArrayScalar_VAL(obj_ptr, UInt64);

        // Stash the memory chunk pointer for later use by boost.python
        data->convertible = storage;
    }
};

/*
 * A silly function to test scalar conversion.
 * The first arg tests automatic function argument conversion.
 * The second arg is used to demonstrate explicit conversion via boost::python::extract<>()
 */
void print_number( uint32_t number, boost::python::object other_number )
{
    using namespace boost::python;
    std::cout << "The number is: " << number << std::endl;
    std::cout << "The other number is: " << extract<int16_t>(other_number) << std::endl;
}

/*
 * Instantiate the python extension module 'printnum'.
 *
 * Example Python usage:
 *
 *     import numpy as np
 *     from printnum import print_number
 *     print_number( np.uint8(123), np.int64(-456) )
 *
 *     ## That prints the following:
 *     # The number is: 123
 *     # The other number is: -456
 */
BOOST_PYTHON_MODULE(printnum)
{
    using namespace boost::python;

    // http://docs.scipy.org/doc/numpy/reference/c-api.array.html#importing-the-api
    import_array();

    // Register conversion for all scalar types.
    NumpyScalarConverter<signed char>();
    NumpyScalarConverter<short>();
    NumpyScalarConverter<int>();
    NumpyScalarConverter<long>();
    NumpyScalarConverter<long long>();
    NumpyScalarConverter<unsigned char>();
    NumpyScalarConverter<unsigned short>();
    NumpyScalarConverter<unsigned int>();
    NumpyScalarConverter<unsigned long>();
    NumpyScalarConverter<unsigned long long>();
    NumpyScalarConverter<float>();
    NumpyScalarConverter<double>();

    // Expose our C++ function as a python function.
    def("print_number", &print_number, (arg("number"), arg("other_number")));
}

Soundboard