Holding python-produced value in a C++ static boost::shared_ptr

Question

While playing with Boost.Python and C++, sometimes we create classes that are bound using the class itself and a boost::shared_ptr<> version. This is very convenient for many reasons and can be used in lots of places. However, the mechanism does not seem to work reliably when boost::python returns a boost::shared_ptr<> to a value that was produced in Python and is recorded on a C++ static variable.

Normally, I'd expect the returned boost::shared_ptr<> to hold a special deleter that would take care of this, but that does not seem to be the case. What seems to happen is that the returned boost::shared_ptr just wraps a pointer to a value produced in Python w/o any special consideration with deletion. This results in a consistent crash coming from a double-deletion (one from the Python interpreter itself and one from the C++ static) - or at least it looks like it.

To reproduce this behavior using the code below, create a test.cc file like below and test with the following script.

#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>

struct A {
  std::string _a;
  A(std::string a): _a(a) {}
  std::string str() { return _a; }
};

static boost::shared_ptr<A> holder(new A("foo"));

static void set_holder(boost::shared_ptr<A> a_ptr) {
  holder = a_ptr;
}

static boost::shared_ptr<A> get_holder() {
  return holder;
}

BOOST_PYTHON_MODULE(test)
{
  using namespace boost::python;

  class_<A, boost::shared_ptr<A> >("A", init<std::string>())
    .def("__str__", &A::str)
    ;

  def("set_holder", &set_holder);
  def("get_holder", &get_holder);
}

With the following Python test program:

import test
print(str(test.get_holder()))
test.set_holder(test.A('bar'))
print(str(test.get_holder()))

Compiling (with g++ -I/usr/include/python2.7 -shared -fpic test.cc -lboost_python -lpython2.7 -o test.so) and running the above program (python test.py) under Linux (ubuntu 12.10, with Python 2.7 and Boost 1.50), resulted in the following stack trace:

#0  0x000000000048aae8 in ?? ()
#1  0x00007fa44f85f589 in boost::python::converter::shared_ptr_deleter::operator()(void const*) () from /usr/lib/libboost_python-py27.so.1.50.0
#2  0x00007fa44fa97cf9 in boost::detail::sp_counted_impl_pd<void*, boost::python::converter::shared_ptr_deleter>::dispose() ()
   from /remote/filer.gx/home.active/aanjos/test.so
#3  0x00007fa44fa93f9c in boost::detail::sp_counted_base::release() ()
   from /remote/filer.gx/home.active/aanjos/test.so
#4  0x00007fa44fa9402b in boost::detail::shared_count::~shared_count() ()
   from /remote/filer.gx/home.active/aanjos/test.so
#5  0x00007fa44fa94404 in boost::shared_ptr<A>::~shared_ptr() ()
   from /remote/filer.gx/home.active/aanjos/test.so
#6  0x00007fa450337901 in __run_exit_handlers (status=0, 
    listp=0x7fa4506b46a8 <__exit_funcs>, run_list_atexit=true) at exit.c:78
#7  0x00007fa450337985 in __GI_exit (status=<optimized out>) at exit.c:100
#8  0x00007fa45031d774 in __libc_start_main (main=0x44b769 <main>, argc=2, 
    ubp_av=0x7fffaa28e2a8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffaa28e298) at libc-start.c:258
#9  0x00000000004ce6dd in _start ()

This indicates a double-deletion has occured at the static destructor. This behavior seems to be consistent among different platforms.

Question: is it possible to achieve the described behavior w/o copying the returned valued from boost::python? In the above toy example, that would be simple, but in my real problem, a deep-copy of A would be impractical.

Edmon

Answer 1

The problem you have is the destruction of the shared_ptr happens after python has finalized. Look at:

• shared_ptr_from_python.hpp
• shared_ptr_deleter
• http://docs.python.org/2/c-api/sys.html#Py_AtExit).

I suggest to encapsulate the shared_ptr, which comes without extra cleanup code. Four soulutions, though:

    #include <boost/python.hpp>
    #include <boost/shared_ptr.hpp>
    #include <iostream>

struct A {
  std::string _a;
  A(std::string a): _a(a) {}
  ~A() { std::cout << "Destruct: " << _a << std::endl; }
  std::string str() { return _a; }
};

void python_exit();

static boost::shared_ptr<A> holder(new A("foo"));

static boost::shared_ptr<A> get_holder() {
  return holder;
}

static void set_holder(boost::shared_ptr<A> a_ptr) {
  // The shared_ptr comes with python::converter::shared_ptr_deleter
  holder = a_ptr;
}


// Fix 1: Cleanup while python is running
// ======================================

void reset_holder() {
   std::cout << "reset" << std::endl;
   holder.reset(new A("holder without shared_ptr_deleter"));
}


// Fix 2: The shared pointer is never deleted (which is a memory leak of a
//        global varialbe). The contained object is destructed, below.
// =========================================================================

static boost::shared_ptr<A>* holder_ptr = new boost::shared_ptr<A>(
    new A("foo_ptr"));

static boost::shared_ptr<A> get_holder_ptr() {
  return *holder_ptr;
}

static void set_holder_ptr(boost::shared_ptr<A> a_ptr) {
  // Note: I know, it's no good to do that here (quick and dirty):
  Py_AtExit(python_exit);
  // The shared_ptr comes with python::converter::shared_ptr_deleter
  *holder_ptr = a_ptr;
}

void python_exit() {
   std::cout << "\n"
     "Since Python’s internal finalization will have completed before the\n"
     "cleanup function, no Python APIs should be called in or after exit.\n"
     "The boost::python::shared_ptr_deleter will do so, though.\n"
     << std::endl;
   // Destruction but no deallocation.
   holder_ptr->get()->~A();
}


// Fix 3: Put a finalizer object into a module.
// =========================================================================

static boost::shared_ptr<A> holder_finalizer(new A("foo_finalizer"));


struct PythonModuleFinalizer
{
    ~PythonModuleFinalizer();
};

PythonModuleFinalizer::~PythonModuleFinalizer() {
    std::cout << "PythonModuleFinalizer" << std::endl;
    holder_finalizer.reset(
        new A("holder_finalizer without shared_ptr_deleter"));
}

static boost::shared_ptr<A> get_holder_finalizer() {
  return holder_finalizer;
}

static void set_holder_finalizer(boost::shared_ptr<A> a_ptr) {
  // The shared_ptr comes with python::converter::shared_ptr_deleter
  holder_finalizer = a_ptr;
}


// Fix 4: Encapsulate the shared_ptr
// =========================================================================

class B {
    private:
    struct I {
        std::string b;
        I(const std::string& b): b(b) {}
        ~I() { std::cout << "Destruct: " << b << std::endl; }
    };

    public:
    B(std::string b): s(new I(b)) {}
    std::string str() { return s.get()->b; }

    private:
    boost::shared_ptr<I> s;
};

static B holder_encapsulate("foo_encapsulate");


static B get_holder_encapsulate() {
  return holder_encapsulate;
}

static void set_holder_encapsulate(B b) {
  holder_encapsulate = b;
}


BOOST_PYTHON_MODULE(test)
{
  using namespace boost::python;

  class_<A, boost::shared_ptr<A> >("A", init<std::string>())
    .def("__str__", &A::str)
    ;

  def("set_holder", &set_holder);
  def("get_holder", &get_holder);
  def("reset_holder", &reset_holder);

  def("set_holder_ptr", &set_holder_ptr);
  def("get_holder_ptr", &get_holder_ptr);

  object finalizer_class = class_<PythonModuleFinalizer
      ("PythonModuleFinalizer", init<>());
  object finalizer = finalizer_class();
  scope().attr("ModuleFinalizer") = finalizer;
  def("set_holder_finalizer", &set_holder_finalizer);
  def("get_holder_finalizer", &get_holder_finalizer);

  class_<B>("B", init<std::string>())
    .def("__str__", &B::str)
  ;
  def("set_holder_encapsulate", &set_holder_encapsulate);
  def("get_holder_encapsulate", &get_holder_encapsulate);

}

The python file:

import test
print(str(test.get_holder()))
test.set_holder(test.A('bar'))
print(str(test.get_holder()))
test.reset_holder()

print(str(test.get_holder_ptr()))
test.set_holder_ptr(test.A('bar_ptr'))
print(str(test.get_holder_ptr()))

print(str(test.get_holder_finalizer()))
test.set_holder_finalizer(test.A('bar_finalizer'))
print(str(test.get_holder_finalizer()))

print(str(test.get_holder_encapsulate()))
test.set_holder_encapsulate(test.B('bar_encapsulate'))
print(str(test.get_holder_encapsulate()))

The output of the test is:

foo
Destruct: foo
bar
reset
Destruct: bar
foo_ptr
Destruct: foo_ptr
bar_ptr
foo_finalizer
Destruct: foo_finalizer
bar_finalizer
foo_encapsulate
Destruct: foo_encapsulate
bar_encapsulate
PythonModuleFinalizer
Destruct: bar_finalizer

Since Python’s internal finalization will have completed before the
cleanup function, no Python APIs should be called in or after exit.
The boost::python::shared_ptr_deleter will do so, though.

Destruct: bar_ptr
Destruct: bar_encapsulate
Destruct: holder without shared_ptr_deleter

Creeper

Thanks for this answer. There is a lot of information here, so I'll take some time to test each of those in my problem. – Theca