Wrapping arrays in Boost Python

Question

I have a series of C++ structures I am trying to wrap using boost python. I've run into difficulties when these structures contain arrays. I am trying to do this with minimal overhead and unfortunately I can't make any modifications to the structs themselves. So for instance say I have

struct Foo
{
    int vals[3];
};

I would like to be able to access this in python as follows:

f = Foo()
f.vals[0] = 10
print f.vals[0]

Right now I am using a series of get/set functions which works but is very inelegant and inconsistent with accessing the other non-array members. Here is my current solution:

int getVals (Foo f, int index) { return f.vals[index]; }
void setVals (Foo& f, int index, int value) { f.vals[index] = value; }

boost::python::class_< Foo > ( "Foo", init<>() )
    .def ( "getVals", &getVals )
    .def ( "setVals", &setVals );

I am fine with having the get/set functions (as there are certain cases where I need to implement a custom get or set operation) but I am not sure how to incorporate the [] operator to access the elements of the array. In other classes which themselves are accessible with the [] operator I have been able to use _getitem_ and _setitem_ which have worked perfectly, but I'm not sure how I would do this with class members if that would even be possible.

Telmatelo

This question may be helpful. – Goose

Answer 1

For such a relatively simple question, the answer becomes rather involved. Before providing the solution, lets first examine the depth of the problem:

 f = Foo()
 f.vals[0] = 10

f.vals returns an intermediate object that provides __getitem__ and __setitem__ methods. For Boost.Python to support this, auxiliary types will need to be exposed for each type of array, and these types will provide indexing support.

One subtle difference between the languages is object lifetime. Consider the following:

 f = Foo()
 v = f.vals
 f = None
 v[0] = 10

With Python object's lifetimes being managed through reference counting, f does not own the object referenced by vals. Hence, even though the object referenced to by f is destroyed when f is set to None, the object referenced by v remains alive. This is on contrast to the C++ Foo type needing to be exposed, as Foo owns the memory to which vals refers. With Boost.Python, the auxiliary object returned by f.vals needs to extend the life of object referenced by f.

---

With the problem examined, lets start on the solution. Here are the basic arrays needing to be exposed:

struct Foo
{
  int vals[3];
  boost::array<std::string, 5> strs;

  Foo()  { std::cout << "Foo()"  << std::endl; }
  ~Foo() { std::cout << "~Foo()" << std::endl; }
};

int more_vals[2];

The auxiliary type for Foo::vals and Foo::strs needs to provide minimal overhead, while supporting indexing. This is accomplished in array_proxy:

/// @brief Type that proxies to an array.
template <typename T>
class array_proxy
{
public:
  // Types
  typedef T           value_type;
  typedef T*          iterator;
  typedef T&          reference;
  typedef std::size_t size_type;

  /// @brief Empty constructor.
  array_proxy()
    : ptr_(0),
      length_(0)
  {}

  /// @brief Construct with iterators.
  template <typename Iterator>
  array_proxy(Iterator begin, Iterator end)
    : ptr_(&*begin),
      length_(std::distance(begin, end))
  {}

  /// @brief Construct with with start and size.
  array_proxy(reference begin, std::size_t length)
    : ptr_(&begin),
      length_(length)
  {}

  // Iterator support.
  iterator begin()               { return ptr_; }
  iterator end()                 { return ptr_ + length_; }

  // Element access.
  reference operator[](size_t i) { return ptr_[i]; }

  // Capacity.
  size_type size()               { return length_; }

private:
  T* ptr_;
  std::size_t length_;
};

With the auxiliary type done, the remaining piece is to add the ability to expose indexing capabilities to the auxiliary type in Python. Boost.Python's indexing_suite provides hooks to add indexing support to exposed types through a policy-based approach. The ref_index_suite class below is a policy class fulfilling the DerivedPolicies type requirements:

/// @brief Policy type for referenced indexing, meeting the DerivedPolicies
///        requirement of boost::python::index_suite.
/// 
/// @note Requires Container to support:
///          - value_type and size_type types,
///          - value_type is default constructable and copyable,
///          - element access via operator[],
///          - Default constructable, iterator constructable,
///          - begin(), end(), and size() member functions
template <typename Container>
class ref_index_suite
  : public boost::python::indexing_suite<Container,
      ref_index_suite<Container> >

{
public:

  typedef typename Container::value_type data_type;
  typedef typename Container::size_type  index_type;
  typedef typename Container::size_type  size_type;

  // Element access and manipulation.

  /// @brief Get element from container.
  static data_type&
  get_item(Container& container, index_type index)
  {
    return container[index];
  }

  /// @brief Set element from container.
  static void
  set_item(Container& container, index_type index, const data_type& value)
  {
    container[index] = value;
  }

  /// @brief Reset index to default value.
  static void
  delete_item(Container& container, index_type index)
  {
    set_item(container, index, data_type());
  };

  // Slice support.

  /// @brief Get slice from container.
  ///
  /// @return Python object containing
  static boost::python::object
  get_slice(Container& container, index_type from, index_type to)
  {
    using boost::python::list;
    if (from > to) return list();

    // Return copy, as container only references its elements.
    list list;
    while (from != to) list.append(container[from++]);
    return list;
  };

  /// @brief Set a slice in container with a given value.
  static void
  set_slice(
    Container& container, index_type from,
    index_type to, const data_type& value
  )
  {
    // If range is invalid, return early.
    if (from > to) return;

    // Populate range with value.
    while (from < to) container[from++] = value;
  }

  /// @brief Set a slice in container with another range.
  template <class Iterator>
  static void
  set_slice(
    Container& container, index_type from,
    index_type to, Iterator first, Iterator last
  )
  {
    // If range is invalid, return early.
    if (from > to) return;

    // Populate range with other range.
    while (from < to) container[from++] = *first++;   
  }

  /// @brief Reset slice to default values.
  static void
  delete_slice(Container& container, index_type from, index_type to)
  {
    set_slice(container, from, to, data_type());
  }

  // Capacity.

  /// @brief Get size of container.
  static std::size_t
  size(Container& container) { return container.size(); }

  /// @brief Check if a value is within the container.
  template <class T>
  static bool
  contains(Container& container, const T& value)
  {
    return std::find(container.begin(), container.end(), value)
        != container.end();
  }

  /// @brief Minimum index supported for container.
  static index_type
  get_min_index(Container& /*container*/)
  {
      return 0;
  }

  /// @brief Maximum index supported for container.
  static index_type
  get_max_index(Container& container)
  {
    return size(container);
  }

  // Misc.

  /// @brief Convert python index (could be negative) to a valid container
  ///        index with proper boundary checks.
  static index_type
  convert_index(Container& container, PyObject* object)
  {
    namespace python = boost::python;
    python::extract<long> py_index(object);

    // If py_index cannot extract a long, then type the type is wrong so
    // set error and return early.
    if (!py_index.check()) 
    {
      PyErr_SetString(PyExc_TypeError, "Invalid index type");
      python::throw_error_already_set(); 
      return index_type();
    }

    // Extract index.
    long index = py_index();

    // Adjust negative index.
    if (index < 0)
        index += container.size();

    // Boundary check.
    if (index >= long(container.size()) || index < 0)
    {
      PyErr_SetString(PyExc_IndexError, "Index out of range");
      python::throw_error_already_set();
    }

    return index;
  }
};

Each auxiliary type needs to be exposed through Boost.Python with boost::python::class_<...>. This can be a bit tedious, so a single auxiliary function will conditionally register types.

/// @brief Conditionally register a type with Boost.Python.
template <typename T>
void register_array_proxy()
{
  typedef array_proxy<T> proxy_type;

  // If type is already registered, then return early.
  namespace python = boost::python;
  bool is_registered = (0 != python::converter::registry::query(
    python::type_id<proxy_type>())->to_python_target_type());
  if (is_registered) return;

  // Otherwise, register the type as an internal type.
  std::string type_name = std::string("_") + typeid(T).name();
  python::class_<proxy_type>(type_name.c_str(), python::no_init)
    .def(ref_index_suite<proxy_type>());
}

Additionally, template argument deduction will be used to provide an easy API to the user:

/// @brief Create a callable Boost.Python object from an array.
template <typename Array>
boost::python::object make_array(Array array)
{
  // Deduce the array_proxy type by removing all the extents from the
  // array.
  ...

  // Register an array proxy.
  register_array_proxy<...>();
}

When being accessed from Python, Foo::vals need to be transformed from int[3] to array_proxy<int>. A template class can serve as a functor that will create array_proxy of the appropriate type. The array_proxy_getter below provides this capability.

/// @brief Functor used used convert an array to an array_proxy for
///        non-member objects.
template <typename NativeType,
          typename ProxyType>
struct array_proxy_getter
{
public:
  typedef NativeType native_type;
  typedef ProxyType  proxy_type;

  /// @brief Constructor.
  array_proxy_getter(native_type array): array_(array) {}

  /// @brief Return an array_proxy for a member array object.
  template <typename C>
  proxy_type operator()(C& c) { return make_array_proxy(c.*array_); }

  /// @brief Return an array_proxy for non-member array object.
  proxy_type operator()() { return make_array_proxy(*array_); }
private:
  native_type array_;
};

Instances of this functor will be wrapped in a callable boost::python::object. The single entry point of make_array is expanded:

/// @brief Create a callable Boost.Python object from an array.
template <typename Array>
boost::python::object make_array(Array array)
{ 
  // Deduce the array_proxy type by removing all the extents from the
  // array.
  ...

  // Register an array proxy.
  register_array_proxy<...>();

  // Create function.
  return boost::python::make_function(
      array_proxy_getter<Array>(array),
      ...);
}

Finally, object lifetime needs to be managed. Boost.Python provides hooks to designate how object lifetimes should be managed through its CallPolices concept. In this case, with_custodian_and_ward_postcall can be used to enforce that the array_proxy<int> returned from foo_vals() extends the lifetime of instance of foo from which it was created.

// CallPolicy type used to keep the owner alive when returning an object
// that references the parents member variable.
typedef boost::python::with_custodian_and_ward_postcall<
    0, // return object (custodian)
    1  // self or this (ward)
  > return_keeps_owner_alive;

---

Below is the complete example supporting non-member and member native and Boost.Array single dimension arrays:

#include <string>
#include <typeinfo>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/indexing_suite.hpp>

namespace detail {

template <typename> struct array_trait;

/// @brief Type that proxies to an array.
template <typename T>
class array_proxy
{
public:
  // Types
  typedef T           value_type;
  typedef T*          iterator;
  typedef T&          reference;
  typedef std::size_t size_type;

  /// @brief Empty constructor.
  array_proxy()
    : ptr_(0),
      length_(0)
  {}

  /// @brief Construct with iterators.
  template <typename Iterator>
  array_proxy(Iterator begin, Iterator end)
    : ptr_(&*begin),
      length_(std::distance(begin, end))
  {}

  /// @brief Construct with with start and size.
  array_proxy(reference begin, std::size_t length)
    : ptr_(&begin),
      length_(length)
  {}

  // Iterator support.
  iterator begin()               { return ptr_; }
  iterator end()                 { return ptr_ + length_; }

  // Element access.
  reference operator[](size_t i) { return ptr_[i]; }

  // Capacity.
  size_type size()               { return length_; }

private:
  T* ptr_;
  std::size_t length_;
};

/// @brief Make an array_proxy.
template <typename T>
array_proxy<typename array_trait<T>::element_type>
make_array_proxy(T& array)
{
  return array_proxy<typename array_trait<T>::element_type>(
    array[0],
    array_trait<T>::static_size);
}

/// @brief Policy type for referenced indexing, meeting the DerivedPolicies
///        requirement of boost::python::index_suite.
/// 
/// @note Requires Container to support:
///          - value_type and size_type types,
///          - value_type is default constructable and copyable,
///          - element access via operator[],
///          - Default constructable, iterator constructable,
///          - begin(), end(), and size() member functions
template <typename Container>
class ref_index_suite
  : public boost::python::indexing_suite<Container,
      ref_index_suite<Container> >
{
public:

  typedef typename Container::value_type data_type;
  typedef typename Container::size_type  index_type;
  typedef typename Container::size_type  size_type;

  // Element access and manipulation.

  /// @brief Get element from container.
  static data_type&
  get_item(Container& container, index_type index)
  {
    return container[index];
  }

  /// @brief Set element from container.
  static void
  set_item(Container& container, index_type index, const data_type& value)
  {
    container[index] = value;
  }

  /// @brief Reset index to default value.
  static void
  delete_item(Container& container, index_type index)
  {
    set_item(container, index, data_type());
  };

  // Slice support.

  /// @brief Get slice from container.
  ///
  /// @return Python object containing
  static boost::python::object
  get_slice(Container& container, index_type from, index_type to)
  {
    using boost::python::list;
    if (from > to) return list();

    // Return copy, as container only references its elements.
    list list;
    while (from != to) list.append(container[from++]);
    return list;
  };

  /// @brief Set a slice in container with a given value.
  static void
  set_slice(
    Container& container, index_type from,
    index_type to, const data_type& value
  )
  {
    // If range is invalid, return early.
    if (from > to) return;

    // Populate range with value.
    while (from < to) container[from++] = value;
  }

  /// @brief Set a slice in container with another range.
  template <class Iterator>
  static void
  set_slice(
    Container& container, index_type from,
    index_type to, Iterator first, Iterator last
  )
  {
    // If range is invalid, return early.
    if (from > to) return;

    // Populate range with other range.
    while (from < to) container[from++] = *first++;   
  }

  /// @brief Reset slice to default values.
  static void
  delete_slice(Container& container, index_type from, index_type to)
  {
    set_slice(container, from, to, data_type());
  }

  // Capacity.

  /// @brief Get size of container.
  static std::size_t
  size(Container& container) { return container.size(); }

  /// @brief Check if a value is within the container.
  template <class T>
  static bool
  contains(Container& container, const T& value)
  {
    return std::find(container.begin(), container.end(), value)
        != container.end();
  }

  /// @brief Minimum index supported for container.
  static index_type
  get_min_index(Container& /*container*/)
  {
      return 0;
  }

  /// @brief Maximum index supported for container.
  static index_type
  get_max_index(Container& container)
  {
    return size(container);
  }

  // Misc.

  /// @brief Convert python index (could be negative) to a valid container
  ///        index with proper boundary checks.
  static index_type
  convert_index(Container& container, PyObject* object)
  {
    namespace python = boost::python;
    python::extract<long> py_index(object);

    // If py_index cannot extract a long, then type the type is wrong so
    // set error and return early.
    if (!py_index.check()) 
    {
      PyErr_SetString(PyExc_TypeError, "Invalid index type");
      python::throw_error_already_set(); 
      return index_type();
    }

    // Extract index.
    long index = py_index();

    // Adjust negative index.
    if (index < 0)
        index += container.size();

    // Boundary check.
    if (index >= long(container.size()) || index < 0)
    {
      PyErr_SetString(PyExc_IndexError, "Index out of range");
      python::throw_error_already_set();
    }

    return index;
  }
};

/// @brief Trait for arrays.
template <typename T>
struct array_trait_impl;

// Specialize for native array.
template <typename T, std::size_t N>
struct array_trait_impl<T[N]>
{
  typedef T element_type;
  enum { static_size = N };
  typedef array_proxy<element_type> proxy_type;
  typedef boost::python::default_call_policies policy;
  typedef boost::mpl::vector<array_proxy<element_type> > signature;
};

// Specialize boost::array to use the native array trait.
template <typename T, std::size_t N>
struct array_trait_impl<boost::array<T, N> >
  : public array_trait_impl<T[N]>
{};

// @brief Specialize for member objects to use and modify non member traits.
template <typename T, typename C>
struct array_trait_impl<T (C::*)>
  : public array_trait_impl<T>
{
  typedef boost::python::with_custodian_and_ward_postcall<
      0, // return object (custodian)
      1  // self or this (ward)
    > policy;

  // Append the class to the signature.
  typedef typename boost::mpl::push_back<
    typename array_trait_impl<T>::signature, C&>::type signature;
};

/// @brief Trait class used to deduce array information, policies, and 
///        signatures
template <typename T>
struct array_trait:
  public array_trait_impl<typename boost::remove_pointer<T>::type>
{
  typedef T native_type;
};

/// @brief Functor used used convert an array to an array_proxy for
///        non-member objects.
template <typename Trait>
struct array_proxy_getter
{
public:
  typedef typename Trait::native_type native_type;
  typedef typename Trait::proxy_type proxy_type;

  /// @brief Constructor.
  array_proxy_getter(native_type array): array_(array) {}

  /// @brief Return an array_proxy for a member array object.
  template <typename C>
  proxy_type operator()(C& c) { return make_array_proxy(c.*array_); }

  /// @brief Return an array_proxy for a non-member array object.
  proxy_type operator()() { return make_array_proxy(*array_); }
private:
  native_type array_;
};

/// @brief Conditionally register a type with Boost.Python.
template <typename Trait>
void register_array_proxy()
{
  typedef typename Trait::element_type element_type;
  typedef typename Trait::proxy_type proxy_type;

  // If type is already registered, then return early.
  namespace python = boost::python;
  bool is_registered = (0 != python::converter::registry::query(
    python::type_id<proxy_type>())->to_python_target_type());
  if (is_registered) return;

  // Otherwise, register the type as an internal type.
  std::string type_name = std::string("_") + typeid(element_type).name();
  python::class_<proxy_type>(type_name.c_str(), python::no_init)
    .def(ref_index_suite<proxy_type>());
}

/// @brief Create a callable Boost.Python object that will return an
///        array_proxy type when called.
///
/// @note This function will conditionally register array_proxy types
///       for conversion within Boost.Python.  The array_proxy will
///       extend the life of the object from which it was called.
///       For example, if `foo` is an object, and `vars` is an array,
///       then the object returned from `foo.vars` will extend the life
///       of `foo`.
template <typename Array>
boost::python::object make_array_aux(Array array)
{
  typedef array_trait<Array> trait_type;
  // Register an array proxy.
  register_array_proxy<trait_type>();

  // Create function.
  return boost::python::make_function(
      array_proxy_getter<trait_type>(array),
      typename trait_type::policy(),
      typename trait_type::signature());
}

} // namespace detail

/// @brief Create a callable Boost.Python object from an array.
template <typename T>
boost::python::object make_array(T array)
{ 
  return detail::make_array_aux(array);
}

struct Foo
{
  int vals[3];
  boost::array<std::string, 5> strs;

  Foo()  { std::cout << "Foo()"  << std::endl; }
  ~Foo() { std::cout << "~Foo()" << std::endl; }
};

int more_vals[2];

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  python::class_<Foo>("Foo")
    .add_property("vals", make_array(&Foo::vals))
    .add_property("strs", make_array(&Foo::strs))
    ;
  python::def("more_vals", make_array(&more_vals));
}

And usage, testing access, slicing, type checking, and lifetime management:

>>> from example import Foo, more_vals
>>> def print_list(l): print ', '.join(str(v) for v in l)
... 
>>> f = Foo()
Foo()
>>> f.vals[0] = 10
>>> print f.vals[0]
10
>>> f.vals[0] = '10'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Invalid assignment
>>> f.vals[100] = 10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: Index out of range
>>> f.vals[:] = xrange(100,103)
>>> print_list(f.vals)
100, 101, 102
>>> f.strs[:] = ("a", "b", "c", "d", "e")
>>> print_list(f.strs)
a, b, c, d, e
>>> f.vals[-1] = 30
>>> print_list(f.vals)
100, 101, 30
>>> v = f.vals
>>> del v[:-1]
>>> print_list(f.vals)
0, 0, 30
>>> print_list(v)
0, 0, 30
>>> x = v[-1:]
>>> f = None
>>> v = None
~Foo()
>>> print_list(x)
30
>>> more_vals()[:] = xrange(50, 100)
>>> print_list(more_vals())
50, 51

Goose

This is very nice! I was stuck in the way to add field "vals" as array and was trying to avoid reimplementing a complete derived policy, but I guess there is no better solution. Why not proposing this implementation for inclusion into Boost? It should be a rather common thing ppl want to achieve. Do you think it is possible to get rid of the array_types::add call ? – Comet

@Raffi: Thanks. I've been working on a few Boost.Python features to propose for inclusion. I am still struggling getting them up to par, but it is a goal. Also, I was able to expand upon the initial solution, removing the need for boilerplate code (such as the array_types::add calls), and provide a convenient make_array() function to support non-member and member native and Boost.Array single dimension arrays. – Goose

@tanner I've been using your solution for a while and it works great for an array of primitives, but when I try exporting an array of structs it doesn't work and I get a compilation error, How can I export an array of structs? – Galimatias

Answer 2

I think it should work easily with the indexing suite of boost.python. For your case, you have to specify the container being a float* and return a constant size in a derived policy.

EDIT: The method above is nice for containers, but hard to use for your case. The simplest thing to do is to declare the two functions set and get:

int getitem_foo(Foo & f, int index)
{
  if(index < 0 || index >=3)
  {
    PyErr_SetString(PyExc_IndexError, "index out of range");
    throw boost::python::error_already_set();;
  }
  return f.vals[index];
}

void setitem_foo(Foo& f, int index, int val)
{
  if(index < 0 || index >=3)
  {
    PyErr_SetString(PyExc_IndexError, "index out of range");
    throw boost::python::error_already_set();;
  }
  f.vals[index] = val;
}

and then:

boost::python::class_< Foo  >("Foo")
  .def("__getitem__", &getitem_foo)
  .def("__setitem__", &setitem_foo)
;

Comet

The problem is there are many other members in the struct, sometimes including other arrays, so I can't have it directly access an array member just by passing an index to the object itself. – Telmatelo

Answer 3

I have try the answer of @Tanner Sansbury above. It is helpful, but there is something bug to fix. Show here.

For example, there is a class such like this:

struct S {
    float a;
    float b;
    bool operator==(const S& s) {
        return a == s.a && b == s.b;
    }
};

class Foo {
 public:
    bool arr[100];
    S brr[100];
}

For user custom type, compile error occurs.

First step, you should implement the operator == function of your custom type, just like above.

Second step, add more code to array_proxy class.

/// @brief Type that proxies to an array.
template <typename T>
class array_proxy {
 public:
    // Types
    typedef std::ptrdiff_t difference_type;
    /// @brief Compare index supported for container.
    static bool
    compare_index(Container& container, index_type a, index_type b) {
        return a < b;
    }
}

Then, it should be ok to compile the code.

Flournoy