How can boost::serialization be used with std::shared_ptr from C++11?

Question

I know that there is a Boost module for serialization of boost::shared_ptr, but I cannot find anything for std::shared_ptr.

Also, I don't know how to implement it easily. I'm afraid that the following code

namespace boost{namespace serialization{
template<class Archive, class T>
inline void serialize(Archive & ar, std::shared_ptr<T> &t, const unsigned int version)
{
  if(Archive::is_loading::value) {T*r;ar>>r;t=r;}
  else {ar<<t.get();}
}
}}//namespaces

doesn't work. Indeed, if some object was referred multiple times, it would be loaded with first run of ar>>r, and after that just a pointer will be copied. However we would create multiple shared_ptr objects pointing to it, and therefore would destruct it more than one time.

Any ideas on that?

Some technical details about the system I'm using:

• OS: Ubuntu 11.10 (x64)
• Compiler: g++ (Ubuntu/Linaro 4.6.1-9ubuntu3) 4.6.1
• boost version: 1.46.1 (installed with sudo apt-get install libboost-dev)

Synagogue

"However we would create multiple shared_ptr objects pointing to it, and therefore would destruct it more than one time." I think that's something that you are expected to deal with in serialization. There's no automated solution that could possibly know that some shared_ptr (or regular pointer for that matter) that was serialized earlier points to the same object. – Emigration

@NicolBolas : One of the stated goals of boost::serialization was "Serialization of STL containers and other commonly used templates". std::shared_ptr is a part of standard library in C++11. That makes me believe that serialization of std::shared_ptr is already implemented somewhere (maybe in boost library (?) or somewhere else), but I just couldn't find it. – Synagogue

C++11, as the name suggests, came out this year. Indeed, the spec was only finalized a few short months ago. And while GCC and Clang support large parts of it, nothing supports everything. Of course it's not fully supported yet; how could you expect it to be? And even then, that doesn't change the fact that it cannot magically know that two things that were serialized at different times actually refer to the same object. This is just as true of naked pointers as smart pointers. It's up to you to handle this conversion. – Emigration

@NicolBolas : Ok, I see that C++11 is new and std::shared_ptr may be not yet supported by boost. However I cannot agree with your second point: a pretty smart serialization of pointers is implemented in boost::serialization and the same concerns boost::shared_ptr. – Synagogue

I'm not at all a boost::serialization expert, but have you tried copying the boost/serialization/shared_ptr.hpp you linked to and replacing all boost::shared_ptr with std::shared_ptr? – Invidious

@NicolBolas No, boost::serialization does indeed manage multiple pointers to one object. Naked and smart pointers. No magic required. – Cletacleti

Answer 1

As of Boost 1.56, the serialization library has built-in support for std::shared_ptr. You do not need to implement your own serialization helper functions if you can use a more recent version of the library.

Anemography

This is the correct answer. It only needs to be complete: Add #include <boost/serialization/shared_ptr.hpp> to use it! – Inoculate

And what about the unique_ptr? Do I need to change all my unique_ptr into shared_ptr? – Interstate

No, you do not need to change unique_ptr; it is supported by boost serialization library. See https://mcmap.net/q/582918/-boost-serialization-std-unique_ptr-support – Anemography

Answer 2

I finally found a solution on how to serialize the std::shared_ptr using boost serialization. All you need is the following piece of code (explanation follows):

#include <boost/serialization/split_free.hpp>
#include <boost/unordered_map.hpp>

//---/ Wrapper for std::shared_ptr<> /------------------------------------------

namespace boost { namespace serialization {

template<class Archive, class Type>
void save(Archive & archive, const std::shared_ptr<Type> & value, const unsigned int /*version*/)
{
    Type *data = value.get();
    archive << data;
}

template<class Archive, class Type>
void load(Archive & archive, std::shared_ptr<Type> & value, const unsigned int /*version*/)
{
    Type *data;
    archive >> data;

    typedef std::weak_ptr<Type> WeakPtr;
    static boost::unordered_map<void*, WeakPtr> hash;

    if (hash[data].expired())
    {
         value = std::shared_ptr<Type>(data);
         hash[data] = value;
    }
    else value = hash[data].lock();
}

template<class Archive, class Type>
inline void serialize(Archive & archive, std::shared_ptr<Type> & value, const unsigned int version)
{
    split_free(archive, value, version);
}

}}

This code simply serializes the object managed by the std::shared_ptr in the function save(). If multiple std::shared_ptr instances point to same object boost serialization will take automatically care to store it only once. The magic happens in load() where boost serialization returns a raw pointer to the object (data). This raw pointer is looked up in a hash that holds a weak_ptr for each raw pointer. In case that the weak_ptr in the hash is expired we can safely create a new shared_ptr instance, let it manage the raw pointer and store a weak_ptr in the hash. In case that the weak_ptr is not expired we simply lock it to return a shared_ptr. This way the reference counting is correct.

Orjonikidze

I think a regular std::map might be better. Also, it's possible for the weak_ptr to expire between checking it and calling lock(). Instead check the result of lock(). Up vote nonetheless. – Pastiche

This code looks strange: archive << data; here you only serialize the pointer and you load it again in load. but where is the data serialized to which datapoints??? and additionally the static hash might map increases till out of memory, there is not deletion of entries... – Nanananak

Are you sure that the hash is necessary? For one thing this is almost a memory leak because hash will keep increasing for even in a long program (?). Secondly I tested the code and simply having value = std::shared_ptr<Type>(data) (instead of the if block) works and (importantly) the deserialized objects points to the same memory, as if the archive already took care of the concurrent pointee. I am missing something? – Tangible

Answer 3

Serialisation is provided by boost and not by the standard library and although shared_ptr is included in the standard it is part of TR1 (technical report 1).

TR1 as of now does not have serialization. So I would recommend that you use boost's shared pointer.

Shambles

Answer 4

You haven't said what "doesn't work" means; it doesn't compile? It doesn't load/store the value properly? It doesn't..what?

There are two problems I can identify here, one may be part of your intentional design though.

The first, you have not made a correct pointer in the load procedure. Let's break it down:

inline void serialize(Archive & ar, std::shared_ptr<T> &t, const unsigned int version) {
    if (1) { //unimportant
        T* r;
        ar >> r;
        t = r;
    }
}

When you make an object of std::shared_ptr, you are instantiating a class template to provide pointer-like capability (as you know). If you made with an int, it will work as an int pointer. However, simply passing the type as T does NOT mean a pointer created of that type will automatically use that template; indeed, you're creating a bare pointer with T* r. It may as well be int *r. You then fail to initialize it with new; r could be pointing anywhere. If it were intialized properly with a new, you MAY get correct reference counting for creation/deletion of that object; this is one area where std::shared_ptr doesn't seem worth the effort to me. I think the assignment from a bare pointer counts as the second reference, not the first, but I may be wrong? Anyhow, that's not the problem. You're probably corrupting the heap; a compiler should spit out a warning about using an uninitialized pointer, it's a wonder it hasn't. I hope you don't have warnings turned off.

If I remember correctly, that declaration of r needs to be replaced with:

std::shared_ptr<T> r = new std::shared_ptr<T>;

Although it may be

std::shared_ptr<T> r = new std::shared_ptr<T>(r());

I haven't used shared_ptr for a while.

TR1, by the way, has been out for at least 2 years. It is based off of boost's shared_ptr. I don't know why you're using Boost 1.46, but I think that it was out by the time shared_ptr became part of the standard? So it should be compatible...?

Anyhow, the second potential error comes with

t = r;

I'm assuming - incorrectly? - that you WISH to decrement the reference count to t by reassigning it (and possibly destroying the object t points to). If you meant to copy it, you would of course use:

*t = *r;

and make sure your copy constructor works properly.

Circuit

Oh, one comment. Boost 1.46 may also not not be compatible with the new C++1x std::shared_ptr, because it differs from std::tr1::shared_ptr, which I think is what Boost 1.46 was probably made to use. – Protozoan

Answer 5

Recent versions of Boost Serialization include support for all standard library smart pointers.

Cavanagh

Answer 6

This is improvement of denim's solution, which supports loading shared_ptr which points to the same memory, but with different types. This problem can appear when archive contains at the same time shared_ptr and shared_ptr which are pointing to the same object, where A is inherited from B.

namespace boost {
namespace serialization {

    template<class Archive, class Type>
    void save(Archive & archive, const std::shared_ptr<Type> & value, const unsigned int /*version*/)
    {
        Type *data = value.get();
        archive << data;
    }

    static std::map<void*, std::weak_ptr<void>> hash;

    template<class Archive, class Type>
    void load(Archive & archive, std::shared_ptr<Type> & value, const unsigned int /*version*/)
    {
        Type *data;
        archive >> data;

        if (hash[data].expired())
        {
            std::shared_ptr<void> ptr(data);
            value = static_pointer_cast<Type>(ptr);
            hash[data] = ptr;
        }
        else value = static_pointer_cast<Type>(hash[data].lock());
    }

    template<class Archive, class Type>
    inline void serialize(Archive & archive, std::shared_ptr<Type> & value, const unsigned int version)
    {
        split_free(archive, value, version);
    }

}}

As a weakness of this realization - one massive map.

Ronnyronsard

Answer 7

This is the result of rolling your own based on the boost shared pointer header e.g. based on <boost/serialization/shared_ptr.hpp>.

Just copy & paste below into a header file and include it:

#ifndef BOOST_SERIALIZATION_STD_SHARED_PTR_HPP
#define BOOST_SERIALIZATION_STD_SHARED_PTR_HPP

// MS compatible compilers support #pragma once
#if defined(_MSC_VER) && (_MSC_VER >= 1020)
# pragma once
#endif

/////////1/////////2/////////3/////////4/////////5/////////6/////////7/////////8
// shared_ptr.hpp: serialization for boost shared pointer

// (C) Copyright 2004 Robert Ramey and Martin Ecker
// Use, modification and distribution is subject to the Boost Software
// License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at
// http://www.boost.org/LICENSE_1_0.txt)

//  See http://www.boost.org for updates, documentation, and revision history.

#include <cstddef> // NULL

#include <boost/config.hpp>
#include <boost/mpl/integral_c.hpp>
#include <boost/mpl/integral_c_tag.hpp>

#include <boost/detail/workaround.hpp>
#include <memory>

#include <boost/serialization/split_free.hpp>
#include <boost/serialization/nvp.hpp>
#include <boost/serialization/version.hpp>
#include <boost/serialization/tracking.hpp>

/////////1/////////2/////////3/////////4/////////5/////////6/////////7/////////8
// shared_ptr serialization traits
// version 1 to distinguish from boost 1.32 version. Note: we can only do this
// for a template when the compiler supports partial template specialization

#ifndef BOOST_NO_TEMPLATE_PARTIAL_SPECIALIZATION
    namespace boost {
    namespace serialization{
        template<class T>
        struct version< ::std::shared_ptr< T > > {
            typedef mpl::integral_c_tag tag;
            #if BOOST_WORKAROUND(__MWERKS__, BOOST_TESTED_AT(0x3206))
            typedef BOOST_DEDUCED_TYPENAME mpl::int_<1> type;
            #else
            typedef mpl::int_<1> type;
            #endif
            #if BOOST_WORKAROUND(__BORLANDC__, BOOST_TESTED_AT(0x570))
            BOOST_STATIC_CONSTANT(int, value = 1);
            #else
            BOOST_STATIC_CONSTANT(int, value = type::value);
            #endif
        };
        // don't track shared pointers
        template<class T>
        struct tracking_level< ::std::shared_ptr< T > > {
            typedef mpl::integral_c_tag tag;
            #if BOOST_WORKAROUND(__MWERKS__, BOOST_TESTED_AT(0x3206))
            typedef BOOST_DEDUCED_TYPENAME mpl::int_< ::boost::serialization::track_never> type;
            #else
            typedef mpl::int_< ::boost::serialization::track_never> type;
            #endif
            #if BOOST_WORKAROUND(__BORLANDC__, BOOST_TESTED_AT(0x570))
            BOOST_STATIC_CONSTANT(int, value = ::boost::serialization::track_never);
            #else
            BOOST_STATIC_CONSTANT(int, value = type::value);
            #endif
        };
    }}
    #define BOOST_SERIALIZATION_SHARED_PTR(T)
#else
    // define macro to let users of these compilers do this
    #define BOOST_SERIALIZATION_SHARED_PTR(T)                         \
    BOOST_CLASS_VERSION(                                              \
        ::std::shared_ptr< T >,                                     \
        1                                                             \
    )                                                                 \
    BOOST_CLASS_TRACKING(                                             \
        ::std::shared_ptr< T >,                                     \
        ::boost::serialization::track_never                           \
    )                                                                 \
    /**/
#endif

namespace boost {
namespace serialization{

#ifndef BOOST_SERIALIZATION_SHARED_PTR_HPP
struct null_deleter {
    void operator()(void const *) const {}
};
#endif

/////////1/////////2/////////3/////////4/////////5/////////6/////////7/////////8
// serialization for shared_ptr

template<class Archive, class T>
inline void save(
    Archive & ar,
    const std::shared_ptr< T > &t,
    const unsigned int /* file_version */
){
    // The most common cause of trapping here would be serializing
    // something like shared_ptr<int>.  This occurs because int
    // is never tracked by default.  Wrap int in a trackable type
    BOOST_STATIC_ASSERT((tracking_level< T >::value != track_never));
    const T * t_ptr = t.get();
    ar << boost::serialization::make_nvp("px", t_ptr);
}

template<class Archive, class T>
inline void load(
    Archive & ar,
    std::shared_ptr< T > &t,
    const unsigned int /*file_version*/
){
    // The most common cause of trapping here would be serializing
    // something like shared_ptr<int>.  This occurs because int
    // is never tracked by default.  Wrap int in a trackable type
    BOOST_STATIC_ASSERT((tracking_level< T >::value != track_never));
    T* r;
    ar >> boost::serialization::make_nvp("px", r);
    ar.reset(t,r);
}

template<class Archive, class T>
inline void serialize(
    Archive & ar,
    std::shared_ptr< T > &t,
    const unsigned int file_version
){
    // correct shared_ptr serialization depends upon object tracking
    // being used.
    BOOST_STATIC_ASSERT(
        boost::serialization::tracking_level< T >::value
        != boost::serialization::track_never
    );
    boost::serialization::split_free(ar, t, file_version);
}

} // namespace serialization
} // namespace boost

#endif // BOOST_SERIALIZATION_STD_SHARED_PTR_HPP

You can view the differences to <boost/serialization/shared_ptr.hpp> here

Basically,

• renamed include guard
• changed boost::shared_ptr to std::shared_ptr
• included <memory> instead of <boost/shared_ptr.hpp>
• protected null_deleter from redefinition in case you also use boost::shared_ptr
• deleted BOOST_SERIALIZATION_SHARED_PTR_132_HPP - whatever that is about?

So far, this seems to be working just fine.

Limpkin

if the boost version you use doesn't support shared_ptr serialization, how can it possibly implement a function ar.reset(t,r); ? – Singletree

I think you misunderstood. This serializes the std::shared_ptr with boost serialization prior to 1.56. Since boost 1.56 boost supports std::shared_ptr itself. – Limpkin

but the archive prior to 1.56 can't have a function ar.reset(t,r); where t is a shared_ptr – Singletree