Boost.Python add bindings to existing PyObject (for exception handling)

Question

In order to expose a C++ exception to Python in a way that actually works, you have to write something like:

std::string scope = py::extract<std::string>(py::scope().attr("__name__"));
std::string full_name = scope + "." + name;
PyObject* exc_type = PyErr_NewException(&full_name[0], PyExc_RuntimeError, 0);
// ...

But this doesn't seem to interract with anything else in Boost.Python. If I want to expose:

struct Error { int code; };

I could write:

py::class_<Error>("Error", py::no_init)
    .def_readonly("code", &Error::code)
;

How can I combine the class binding for Error with the exception creation on PyErr_NewException? Basically, I want to throw Error{42} and have that work in the obvious way from Python : I can catch by Error or RuntimeError and have that work, and I can catch by AssertionError (or similar) and have that neither catch the Error nor throw a SystemError.

Rubbing

Slightly off topic suggestion but, Cython seems to handle this nicely . Maybe you could wrap most of the C++ code with boost-python and handle such tricky cases with more flexible tools like Cython? – Mincey

Answer 1

The Python type created with class_ has an incompatible layout with Python exceptions types. Attempting to create a type containing both in its hierarchy will fail with a TypeError. As the Python except clause will perform type checking, one option is to create a Python exception type that:

• derives from the desired Python exception type(s)
• proxies to an embedded subject object that is an instance of a type exposed through Boost.Python

This approach requires a few steps:

• create a Python Exception type, deriving from Python Exceptions
• modify the user-defined Python exception's __delattr__, __getattr__ and __setattr methods so that they proxy to an embedded subject object
• patch the user-defined Python exception's initializer to embed a subject object to which it will proxy

A pure Python implementation of the approach would be as follows:

def as_exception(base):
    ''' Decorator that will return a type derived from `base` and proxy to the
       decorated class.

    '''
    def make_exception_type(wrapped_cls):
        # Generic proxying to subject.
        def del_subject_attr(self, name):
            return delattr(self._subject, name)

        def get_subject_attr(self, name):
            return getattr(self._subject, name)

        def set_subject_attr(self, name, value):
            return setattr(self._subject, name, value)

        # Create new type that derives from base and proxies to subject.
        exception_type = type(wrapped_cls.__name__, (base,), {
           '__delattr__': del_subject_attr,
           '__getattr__': get_subject_attr,
           '__setattr__': set_subject_attr,
        })

        # Monkey-patch the initializer now that it has been created.
        original_init = exception_type.__init__

        def init(self, *args, **kwargs):
            original_init(self, *args, **kwargs)
            self.__dict__['_subject'] = wrapped_cls(*args, **kwargs)
        exception_type.__init__ = init

        return exception_type
    return make_exception_type


@as_exception(RuntimeError)
class Error:
    def __init__(self, code):
        self.code = code

assert(issubclass(Error, RuntimeError))
try:
    raise Error(42)
except RuntimeError as e:
    assert(e.code == 42)
except:
    assert(False)

The same general approach can be used by Boost.Python, obviating the need to write the equivalent of class_ for exceptions. However, there are additional steps and considerations:

• register a translator with boost::python::register_exception_translator() that will construct the user-defined Python exception when an instance of the C++ object is thrown
• the subject type may not have an exposed initializer to Python. Hence, when creating an instance of the exception in Python, one should attempt to initialize the subject with __init__. On the other hand, when creating an instance of the exception in C++, one should use a to-python conversion as to avoid __init__.
• one may wish to register from-Python converters to allow an instance of the exception type to be passed from Python to C++, converting it to an instance of the wrapped subject.

---

Below is a complete example demonstrating the approach described above:

#include <boost/python.hpp>

namespace exception {
namespace detail {

/// @brief Return a Boost.Python object given a borrowed object.
template <typename T>
boost::python::object borrowed_object(T* object)
{
  namespace python = boost::python;
  python::handle<T> handle(python::borrowed(object));
  return python::object(handle);
}

/// @brief Return a tuple of Boost.Python objects given borrowed objects.
boost::python::tuple borrowed_objects(
  std::initializer_list<PyObject*> objects)
{
  namespace python = boost::python;
  python::list objects_;

  for(auto&& object: objects)
  {
    objects_.append(borrowed_object(object));
  }

  return python::tuple(objects_);
}

/// @brief Get the class object for a wrapped type that has been exposed
///        through Boost.Python.
template <typename T>
boost::python::object get_instance_class()
{
  namespace python = boost::python;
  python::type_info type = python::type_id<T>();
  const python::converter::registration* registration =
    python::converter::registry::query(type);

  // If the class is not registered, return None.
  if (!registration) return python::object();

  return detail::borrowed_object(registration->get_class_object());
}

} // namespace detail
namespace proxy {

/// @brief Get the subject object from a proxy.
boost::python::object get_subject(boost::python::object proxy)
{
  return proxy.attr("__dict__")["_obj"];
}

/// @brief Check if the subject has a subject.
bool has_subject(boost::python::object proxy)
{
  return boost::python::extract<bool>(
    proxy.attr("__dict__").attr("__contains__")("_obj"));
}

/// @brief Set the subject object on a proxy object.
boost::python::object set_subject(
  boost::python::object proxy,
  boost::python::object subject)
{
  return proxy.attr("__dict__")["_obj"] = subject;
}

/// @brief proxy's __delattr__ that delegates to the subject.
void del_subject_attr(
  boost::python::object proxy,
  boost::python::str name)
{
  delattr(get_subject(proxy), name);
};

/// @brief proxy's __getattr__ that delegates to the subject.
boost::python::object get_subject_attr(
  boost::python::object proxy,
  boost::python::str name)
{
  return getattr(get_subject(proxy), name);
};

/// @brief proxy's __setattr__ that delegates to the subject.
void set_subject_attr(
  boost::python::object proxy,
  boost::python::str name,
  boost::python::object value)
{
  setattr(get_subject(proxy), name, value);
};

boost::python::dict proxy_attrs()
{
  // By proxying to Boost.Python exposed object, one does not have to
  // reimplement the entire Boost.Python class_ API for exceptions.

  // Generic proxying.
  boost::python::dict attrs;
  attrs["__detattr__"] = &del_subject_attr;
  attrs["__getattr__"] = &get_subject_attr;
  attrs["__setattr__"] = &set_subject_attr;
  return attrs;
}

} // namespace proxy

/// @brief Registers from-Python converter for an exception type.
template <typename Subject>
struct from_python_converter
{
  from_python_converter()
  {
    boost::python::converter::registry::push_back(
      &convertible,
      &construct,
      boost::python::type_id<Subject>()
    );
  }

  static void* convertible(PyObject* object)
  {
    namespace python = boost::python;
    python::object subject = proxy::get_subject(
      detail::borrowed_object(object)
    );

    // Locate registration based on the C++ type.
    python::object subject_instance_class =
      detail::get_instance_class<Subject>();
    if (!subject_instance_class) return nullptr;

    bool is_instance = (1 == PyObject_IsInstance(
      subject.ptr(),
      subject_instance_class.ptr()
    ));
    return is_instance
       ? object
       : nullptr;
  }

  static void construct(
    PyObject* object,
    boost::python::converter::rvalue_from_python_stage1_data* data)
  {
    // Object is a borrowed reference, so create a handle indicting it is
    // borrowed for proper reference counting.
    namespace python = boost::python;
    python::object proxy = detail::borrowed_object(object);

    // Obtain a handle to the memory block that the converter has allocated
    // for the C++ type.
    using storage_type =
      python::converter::rvalue_from_python_storage<Subject>;
    void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

    // Copy construct the subject into the converter storage block.
    python::object subject = proxy::get_subject(proxy);
    new (storage) Subject(python::extract<const Subject&>(subject)());

    // Indicate the object has been constructed into the storage.
    data->convertible = storage;
  }

};

/// @brief Expose an exception type in the current scope, that embeds and
//         proxies to the Wrapped type.
template <typename Wrapped>
class exception:
  boost::python::object
{
public:

  /// @brief Expose a RuntimeError exception type with the provided name.
  exception(const char* name) : exception(name, {}) {}

  /// @brief Expose an expcetion with the provided name, deriving from the
  ///        borrowed base type.
  exception(
    const char* name,
    PyObject* borrowed_base
  ) : exception(name, {borrowed_base}) {}

  /// @brief Expose an expcetion with the provided name, deriving from the
  ///        multiple borrowed base type.
  exception(
    const char* name,
    std::initializer_list<PyObject*> borrowed_bases
  ) : exception(name, detail::borrowed_objects(borrowed_bases)) {}

  /// @brief Expose an expcetion with the provided name, deriving from tuple
  ///        of bases.
  exception(
    const char* name,
    boost::python::tuple bases)
  {
    // Default to deriving from Python's RuntimeError.
    if (!bases)
    {
      bases = make_tuple(detail::borrowed_object(PyExc_RuntimeError));
    }

    register_exception_type(name, bases);
    patch_initializer();
    register_translator();
  }

public:

  exception& enable_from_python()
  {
    from_python_converter<Wrapped>{};
    return *this;
  }

private:

  /// @brief Handle to this class object.
  boost::python::object this_class_object() { return *this; }

  /// @brief Create the Python exception type and install it into this object.
  void register_exception_type(
    std::string name,
    boost::python::tuple bases)
  {
    // Copy the instance class' name and scope.
    namespace python = boost::python;
    auto scoped_name = python::scope().attr("__name__") + "." + name;

    // Docstring handling.
    auto docstring = detail::get_instance_class<Wrapped>().attr("__doc__");

    // Create exception dervied from the desired exception types, but with
    // the same name as the Boost.Python class.  This is required because
    // Python exception types and Boost.Python classes have incompatiable
    // layouts.
    // >> type_name = type(fullname, (bases,), {proxying attrs})
    python::handle<> handle(PyErr_NewExceptionWithDoc(
      python::extract<char*>(scoped_name)(),
      docstring ? python::extract<char*>(docstring)() : nullptr,
      bases.ptr(),
      proxy::proxy_attrs().ptr()
    ));

    // Assign the exception type to this object.
    python::object::operator=(python::object{handle});

    // Insert this object into current scope.
    setattr(python::scope(), name, this_class_object());
  }

  /// @brief Patch the initializer to install the delegate object.
  void patch_initializer()
  {
    namespace python = boost::python;
    auto original_init = getattr(this_class_object(), "__init__");

    // Use raw function so that *args and **kwargs can transparently be
    // passed to the initializers.
    this_class_object().attr("__init__") = python::raw_function(
      [original_init](
        python::tuple args,  // self + *args
        python::dict kwargs) // **kwargs
      {
        original_init(*args, **kwargs);
        // If the subject does not exists, then create it.
        auto self = args[0];
        if (!proxy::has_subject(self))
        {
          proxy::set_subject(self, detail::get_instance_class<Wrapped>()(
            *args[python::slice(1, python::_)], // args[1:]
            **kwargs
          ));
        }

        return python::object{}; // None
      });
  }

  // @brief Register translator within the Boost.Python exception handling
  //        chaining.  This allows for an instance of the wrapped type to be
  //        converted to an instance of this exception.
  void register_translator()
  {
    namespace python = boost::python;
    auto exception_type = this_class_object();
    python::register_exception_translator<Wrapped>(
      [exception_type](const Wrapped& proxied_object)
      {
        // Create the exception object.  If a subject is not installed before
        // the initialization of the instance, then a subject will attempt to
        // be installed.  As the subject may not be constructible from Python,
        // manually inject a subject after construction, but before
        // initialization.
        python::object exception_object = exception_type.attr("__new__")(
          exception_type
        );

        proxy::set_subject(exception_object, python::object(proxied_object));

        // Initialize the object.
        exception_type.attr("__init__")(exception_object);

        // Set the exception.
        PyErr_SetObject(exception_type.ptr(), exception_object.ptr());
      });
  }
};

// @brief Visitor that will turn the visited class into an exception,
// /      enabling exception translation.
class export_as_exception
  : public boost::python::def_visitor<export_as_exception>
{
public:

  /// @brief Expose a RuntimeError exception type.
  export_as_exception() : export_as_exception({}) {}

  /// @brief Expose an expcetion type deriving from the borrowed base type.
  export_as_exception(PyObject* borrowed_base)
    : export_as_exception({borrowed_base}) {}

  /// @brief Expose an expcetion type deriving from multiple borrowed
  ///        base types.
  export_as_exception(std::initializer_list<PyObject*> borrowed_bases)
    : export_as_exception(detail::borrowed_objects(borrowed_bases)) {}

  /// @brief Expose an expcetion type deriving from multiple bases.
  export_as_exception(boost::python::tuple bases) : bases_(bases) {}

private:

  friend class boost::python::def_visitor_access;

  template <typename Wrapped, typename ...Args>
  void visit(boost::python::class_<Wrapped, Args...> instance_class) const
  {
    exception<Wrapped>{
      boost::python::extract<const char*>(instance_class.attr("__name__"))(),
      bases_
    };
  }

private:
  boost::python::tuple bases_;
};

} // namespace exception

struct foo { int code; };

struct spam
{
  spam(int code): code(code) {}
  int code;
};

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Expose `foo` as `example.FooError`.
  python::class_<foo>("FooError", python::no_init)
    .def_readonly("code", &foo::code)
    // Redefine the exposed `example.FooError` class as an exception.
    .def(exception::export_as_exception(PyExc_RuntimeError));
    ;

  // Expose `spam` as `example.Spam`.
  python::class_<spam>("Spam", python::init<int>())
    .def_readwrite("code", &spam::code)
    ;

  // Also expose `spam` as `example.SpamError`.
  exception::exception<spam>("SpamError", {PyExc_IOError, PyExc_SystemError})
    .enable_from_python()
    ;

  // Verify from-python.
  python::def("test_foo", +[](int x){ throw foo{x}; });
  // Verify to-Python and from-Python.
  python::def("test_spam", +[](const spam& error) { throw error; });
}

In the above example, the C++ foo type is exposed as example.FooError, then example.FooError gets redefined to an exception type that derives from RuntimeError and proxies to the original example.FooError. Additionally, the C++ spam type is exposed as example.Spam, and an exception type example.SpamError is defined that derives from IOError and SystemError, and proxies to example.Spam. The example.SpamError is also convertible to the C++ spam type.

Interactive usage:

>>> import example
>>> try:
...     example.test_foo(100)
... except example.FooError as e:
...     assert(isinstance(e, RuntimeError))
...     assert(e.code == 100)
... except:
...     assert(False)
...
>>> try:
...     example.test_foo(101)
... except RuntimeError as e:
...     assert(isinstance(e, example.FooError))
...     assert(e.code == 101)
... except:
...     assert(False)
...
... spam_error = example.SpamError(102)
... assert(isinstance(spam_error, IOError))
... assert(isinstance(spam_error, SystemError))
>>> try:
...     example.test_spam(spam_error)
... except IOError as e:
...     assert(e.code == 102)
... except:
...     assert(False)
...

Hilly

This is pretty amazing. So basically, having a C++ exception hierarchy is out since there's really no way to express that in Python via Boost? – Rubbing