How to hook up Boost serialization & iostreams to serialize & gzip an object to string?

Question

I've been using the Boost serialization library, which is actually pretty nice, and lets me make simple wrappers to save my serializable objects to strings, like so:

namespace bar = boost::archive;
namespace bio = boost::iostreams;

template <class T> inline std::string saveString(const T & o) {
 std::ostringstream oss;
 bar::binary_oarchive oa(oss);
 oa << o;
 return oss.str();
}
template <class T> inline void saveFile(const T & o, const char* fname) {
 std::ofstream ofs(fname, std::ios::out|std::ios::binary|std::ios::trunc);
 bar::binary_oarchive oa(ofs);
 oa << o;
}
template <class T> inline void loadFile(T & o, const char* fname) {
 std::ifstream ifs(fname, std::ios::in|std::ios::binary);
 assert(ifs.good()); // XXX catch if file not found
 bar::binary_iarchive ia(ifs);
 ia >> o;
}

The thing is, I just found the need to compress my serialized data, too, so I'm looking at doing that with the filters in boost::iostreams. I figured out how to do it successfully with files:

template <class T> inline void saveGZFile(const T & o, const char* fname) {
 std::ofstream ofs(fname, std::ios::out|std::ios::binary|std::ios::trunc);
 bio::filtering_streambuf<bio::output> out;
 out.push(boost::iostreams::gzip_compressor());
 out.push(ofs);
 bar::binary_oarchive oa(out);
 oa << o;
}
template <class T> inline void loadGZFile(T & o, const char* fname) {
 std::ifstream ifs(fname, std::ios::in|std::ios::binary);
 assert(ifs.good()); // XXX catch if file not found
 bio::filtering_streambuf<bio::input> in;
 in.push(bio::gzip_decompressor());
 in.push(ifs);
 bar::binary_iarchive ia(in);
 ia >> o;
}

But can't figure out how to save correctly to a compressed string. The problem is that I'm not flushing the chain of filters, but I've tried popping and syncing and nothing seems to work. Here's my broken code:

template <class T> inline std::string saveGZString(const T & o) {
 std::ostringstream oss;
 bio::filtering_streambuf<bio::output> out;
 out.push(bio::gzip_compressor());
 out.push(oss);
 bar::binary_oarchive oa(out);
 oa << o;
 // XXX out.pop() twice?  out.strict_sync()??  oss.flush()??
 return oss.str();
}

As a result some data gets stuck in the stream buffer somewhere, and I always end up with a a few complete blocks (16K or 32K) of compressed data when I know it should be 43K or so given the (valid) output I get from using my saveGZFile method. Apparently hooking up the ofstream closes and flushes properly, but hooking up the ostringstream doesn't.

Any help? (This is my first stackoverflow question — help me, guys, you're my only hope!)

Tuatara

Answer 1

Returning to this question, I realized I must've fixed it sometime last year (as I'm using saveGZString right now). Digging to see how I fixed it, it was pretty silly/simple:

namespace bar = boost::archive;
namespace bio = boost::iostreams;

template <typename T> inline std::string saveGZString(const T & o) {
        std::ostringstream oss;
        { 
                bio::filtering_stream<bio::output> f;
                f.push(bio::gzip_compressor());
                f.push(oss);
                bar::binary_oarchive oa(f);
                oa << o;
        } // gzip_compressor flushes when f goes out of scope
        return oss.str();
}

Just let the whole chain go out of scope and it works! Neat! Here's my loader for completeness:

template <typename T> inline void loadGZString(T & o, const std::string& s) {
        std::istringstream iss(s);
        bio::filtering_stream<bio::input> f;
        f.push(bio::gzip_decompressor());
        f.push(iss);
        bar::binary_iarchive ia(f);
        ia >> o;
}

Tuatara

You can avoid the scope limitation trick with a call to flush(). f.flush() – Payne

in the non-working code in my question, the comment says "oss.flush()??" because calling flush() on the ostringstream wasn't working. the scope limitation trick is the only thing that worked for me. unless you mean i should flush f, which doesn't have a flush method (it has a strict_sync() method that is supposed to call flush() on every device in the pipeline, which also I tried, to no avail). – Tuatara

Thanks for this--I was running into the same problem. No flush available and strict_sync didn't have an effect. – Backsaw

Answer 2

I haven't run the code myself, but my best guess is to use out.strict_sync() which applies flush() to every filter/device in the pipeline. I can't seem to tell, though, if gzip_compressor is flushable. If it is not, then strict_sync() will return false, and sync() would be more appropriate.

Metcalf

Internally, it is flush. But, it is flush applied to every filter/device in the chain. – Metcalf

as I recall (a long time ago) I tried strict_sync() and it didn't work, as well as the other things in the frustrated // XXX ?? line in my question ... maybe it works in the latest Boost, who knows. – Tuatara

Compressors are not really flushable. Thus, if a compressor exists in the chain, strict_sync() always returns false. Surprisingly, getting the compressor via auto* ptr = f.component<bio::gzip_compressor>(0) and then calling boost::iostreams::flush(*p) returns true, but does not have an effect. (That's what I meant with "not really flushable.") – Unsling