Boost::Python, converting tuple to Python works, vector<tuple> does not

Question

I've been using Boost::Python for a while, and everything always turned out ok. However yesterday I was trying to find out why a particular type I thought I had registered (a tuple) was giving me errors when I was trying to access it from Python.

Turns out that while the tuple was actually registered, when trying to access it through an std::vector wrapped via the vector_indexing_suite this is not enough anymore.

I was wondering, why is it not working? Is there any way to make this work? Should I try to wrap the vector by hand?

Below is my MVE:

#include <tuple>
#include <vector>

#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>

template <typename T>
struct TupleToPython {
    TupleToPython() {
        boost::python::to_python_converter<T, TupleToPython<T>>();
    }

    template<int...>
    struct sequence {};

    template<int N, int... S>
    struct generator : generator<N-1, N-1, S...> { };

    template<int... S>
    struct generator<0, S...> {
        using type = sequence<S...>;
    };

    template <int... I>
    static boost::python::tuple boostConvertImpl(const T& t, sequence<I...>) {
        return boost::python::make_tuple(std::get<I>(t)...);
    }

    template <typename... Args>
    static boost::python::tuple boostConvert(const std::tuple<Args...> & t) {
        return boostConvertImpl(t, typename generator<sizeof...(Args)>::type());
    }

    static PyObject* convert(const T& t) {
        return boost::python::incref(boostConvert(t).ptr());
    }
};

using MyTuple = std::tuple<int>;
using Tuples = std::vector<MyTuple>;

MyTuple makeMyTuple() {
    return MyTuple();
}

Tuples makeTuples() {
    return Tuples{MyTuple()};
}

BOOST_PYTHON_MODULE(h)
{
    using namespace boost::python;

    TupleToPython<MyTuple>();
    def("makeMyTuple", makeMyTuple);

    class_<std::vector<MyTuple>>{"Tuples"}
        .def(vector_indexing_suite<std::vector<MyTuple>>());
    def("makeTuples", makeTuples);
}

Accessing the resulting .so via Python results in:

>>> print makeMyTuple()
(0,)
>>> print makeTuples()[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No Python class registered for C++ class std::tuple<int>
>>> 

EDIT: I've realized that the error does not happen if the vector_indexing_suite is used with the NoProxy parameter set to true. However, I'd prefer if this wasn't necessary, as it makes the exported classes unintuitive in Python.

Preciado

shouldn't Tuples makeTuples() { return Tuples{MyTuple()}; } be Tuples makeTuples() { return Tuples(); } instead ? – Contrabandist

@Contrabandist I did that so that you could call makeTuples()[0] without triggering the out_of_bounds error, since then the vector would be empty and you wouldn't see the tuple error. – Preciado

Answer 1

TupleToPython registers C++-to-Python converters and Python-to-C++ converters. This is fine.

On the other hand, you want your vector elements to be returned by reference. But there's nothing on the Python side that can serve as a reference to your tuple. A converted-to-Python tuple may hold the same values, but it is completely detached from the original C++ tuple.

It looks like in order to export a tuple by reference, one would need to create an indexing suite for it, rather than to/from-Python converters. I have never done that and cannot guarantee it will work.

Here's how one could expose a tuple as a minimal tuple-like Python object (with only len() and indexing). First define some helper functions:

template <typename A>
int tuple_length(const A&)
{
    return std::tuple_size<A>::value;
}

template <int cidx, typename ... A>
typename std::enable_if<cidx >= sizeof...(A), boost::python::object>::type
get_tuple_item_(const std::tuple<A...>& a, int idx, void* = nullptr)
{
    throw std::out_of_range{"Ur outta range buddy"};
}

template <int cidx, typename ... A, typename = std::enable_if<(cidx < sizeof ...(A))>>
typename std::enable_if<cidx < sizeof...(A), boost::python::object>::type
get_tuple_item_(const std::tuple<A...>& a, int idx, int = 42)
{
    if (idx == cidx)
        return boost::python::object{std::get<cidx>(a)};
    else
        return get_tuple_item_<cidx+1>(a, idx);
};

template <typename A>
boost::python::object get_tuple_item(const A& a, int index)
{
    return get_tuple_item_<0>(a, index);
}

Then expose specific tuples:

using T1 = std::tuple<int, double, std::string>;
using T2 = std::tuple<std::string, int>;

BOOST_PYTHON_MODULE(z)
{
    using namespace boost::python;

    class_<T1>("T1", init<int, double, std::string>())
      .def("__len__", &tuple_length<T1>)
      .def("__getitem__", &get_tuple_item<T1>);

    class_<T2>("T2", init<std::string, int>())
      .def("__len__", &tuple_length<T2>)
      .def("__getitem__", &get_tuple_item<T2>);
}

Note these quasi-tuples, unlike real Python tuples, are mutable (via C++). Because of tuple immutability, exporting via converters and NoProxy looks like a viable alternative to this.

Luteolin

Boost already handles arbitrary classes by reference, though. Why is a tuple any different? – Preciado

It does, sure, if you export your class. Notice how you have to export the vector with class_<std::vector<MyTuple>>{"Tuples"}. – Luteolin

So basically the conversion functions are useful only when you need to pass something by value, right? One would have to create a template class taking a tuple type and registering a class_ with the same interface as a native Python tuple, and then it would work. Correct? – Preciado

Yes this seems to be correct. I'm not quite sure how to register __len__ and __getitem__ correctly for std::tuple but if you manage this it looks like the way to go. – Luteolin

@Preciado I updated the answer with some working code. – Luteolin