boost serialization vs google protocol buffers? [closed]

Asked 30/6, 2009 at 0:2 Answered 14/1, 2018 at 8:31

Solved c++performance serialization protocol-buffers boost-serialization

Does anyone with experience with these libraries have any comment on which one they preferred? Were there any performance differences or difficulties in using?

Glossy answered 30/6, 2009 at 0:2 Comment(1)

This is close to a duplicate of https://mcmap.net/q/281639/-c-serialization-performance so it might be helpful as well – Entomophagous 7/9, 2009 at 17:5

I've played around a little with both systems, nothing serious, just some simple hackish stuff, but I felt that there's a real difference in how you're supposed to use the libraries.

With boost::serialization, you write your own structs/classes first, and then add the archiving methods, but you're still left with some pretty "slim" classes, that can be used as data members, inherited, whatever.

With protocol buffers, the amount of code generated for even a simple structure is pretty substantial, and the structs and code that's generated is more meant for operating on, and that you use protocol buffers' functionality to transport data to and from your own internal structures.

Venola answered 30/6, 2009 at 12:12 Comment(1)

Very often, you don't need to transport data to and from your own internal structures - you can populate those structures directly with protocol buffers. They're essentially value classes, they have move semantics, and they play nicely with STL and Boost collections. – Samovar 7/5, 2014 at 18:47

I've been using Boost Serialization for a long time and just dug into protocol buffers, and I think they don't have the exact same purpose. BS (didn't see that coming) saves your C++ objects to a stream, whereas PB is an interchange format that you read to/from.

PB's datamodel is way simpler: you get all kinds of ints and floats, strings, arrays, basic structure and that's pretty much it. BS allows you to directly save all of your objects in one step.

That means with BS you get more data on the wire but you don't have to rebuild all of your objects structure, whereas protocol buffers is more compact but there is more work to be done after reading the archive. As the name says, one is for protocols (language-agnostic, space efficient data passing), the other is for serialization (no-brainer objects saving).

So what is more important to you: speed/space efficiency or clean code?

Jaconet answered 28/3, 2010 at 11:9 Comment(0)

I've played around a little with both systems, nothing serious, just some simple hackish stuff, but I felt that there's a real difference in how you're supposed to use the libraries.

Venola answered 30/6, 2009 at 12:12 Comment(1)

There are a couple of additional concerns with boost.serialization that I'll add to the mix. Caveat: I don't have any direct experience with protocol buffers beyond skimming the docs.

Note that while I think boost, and boost.serialization, is great at what it does, I have come to the conclusion that the default archive formats it comes with are not a great choice for a wire format.

It's important to distinguish between versions of your class (as mentioned in other answers, boost.serialization has some support for data versioning) and compatibility between different versions of the serialization library.

Newer versions of boost.serialization may not generate archives that older versions can deserialize. (the reverse is not true: newer versions are always intended to deserialize archives made by older versions). This has led to the following problems for us:

Both our client & server software create serialized objects that the other consumes, so we can only move to a newer boost.serialization if we upgrade both client and server in lockstep. (This is quite a challenge in an environment where you don't have full control of your clients).
Boost comes bundled as one big library with shared parts, and both the serialization code and the other parts of the boost library (e.g. shared_ptr) may be in use in the same file, I can't upgrade any parts of boost because I can't upgrade boost.serialization. I'm not sure if it's possible/safe/sane to attempt to link multiple versions of boost into a single executable, or if we have the budget/energy to refactor out bits that need to remain on an older version of boost into a separate executable (DLL in our case).
The old version of boost we're stuck on doesn't support the latest version of the compiler we use, so we're stuck on an old version of the compiler too.

Google seem to actually publish the protocol buffers wire format, and Wikipedia describes them as forwards-compatible, backwards-compatible (although I think Wikipedia is referring to data versioning rather than protocol buffer library versioning). Whilst neither of these is a guarantee of forwards-compatibility, it seems like a stronger indication to me.

In summary, I would prefer a well-known, published wire format like protocol buffers when I don't have the ability to upgrade client & server in lockstep.

Footnote: shameless plug for a related answer by me.

Jarlen answered 25/11, 2011 at 19:39 Comment(3)

FYI: 2.5 years on, I now have plenty of direct experience with protocol buffers, and I stand by my remarks above. – Jarlen 25/7, 2014 at 8:49

What does mean "old version of boost we're stuck on doesn't support the latest version of the compiler we use"? Does Boost use some non-standard language extensions, or why did that happen? – Otolith 7/5, 2018 at 9:50

Good question. I think yes — eg I’d guess that boost.thread and boost.interprocess use (or used) platform-specific APIs to implement features that may not be possible to implement portably. Also the standards conformance of compilers varies across versions, so something that worked with e.g. MSVC 2005 may not work/compile with later versions. Boost.org documents which compilers are known to work: boost.org/doc/libs/1_67_0/libs/log/doc/html/log/… – Jarlen 8/5, 2018 at 7:3

Boost Serialisation

is a library for writing data into a stream.
does not compress data.
does not support data versioning automatically.
supports STL containers.
properties of data written depend on streams chosen (e.g. endian, compressed).

Protocol Buffers

generates code from interface description (supports C++, Python and Java by default. C, C# and others by 3rd party).
optionally compresses data.
handles data versioning automatically.
handles endian swapping between platforms.
does not support STL containers.

Boost serialisation is a library for converting an object into a serialised stream of data. Protocol Buffers do the same thing, but also do other work for you (like versioning and endian swapping). Boost serialisation is simpler for "small simple tasks". Protocol Buffers are probably better for "larger infrastructure".

EDIT:24-11-10: Added "automatically" to BS versioning.

Semaphore answered 7/6, 2010 at 14:2 Comment(5)

Hmm ... this seems not to be exactly correct as BS has versioning and as far as I know endian issues depend on which serialization format you use. – Chrissychrist 24/11, 2010 at 13:18

Ok, it doesn't support is automatically. You have to explicitly build versioning into your serialisation code, the serialiser won't do it for you. – Semaphore 24/11, 2010 at 13:49

boost serialization has included versioning of data from the beginning. It's the default and automatic. If you don't want some data type to be versioned, you can disable it on a type by type basis. But generally this is almost never done. – Imray 7/2, 2014 at 22:5

Data compression is generally done by composition with boost iostream. It's not part of boost serialization because doing so would be redundant. – Imray 7/2, 2014 at 22:6

Data compression with boost is very very easy to add to the mix: #1753969 – Telfore 14/11, 2014 at 12:37

I have no experience with boost serialization, but I have used protocol buffers. I like protocol buffers a lot. Keep the following in mind (I say this with no knowledge of boost).

Protocol buffers are very efficient so I don't imagine that being a serious issue vs. boost.
Protocol buffers provide an intermediate representation that works with other languages (Python and Java... and more in the works). If you know you're only using C++, maybe boost is better, but the option to use other languages is nice.
Protocol buffers are more like data containers... there is no object oriented nature, such as inheritance. Think about the structure of what you want to serialize.
Protocol buffers are flexible because you can add "optional" fields. This basically means you can change the structure of protocol buffer without breaking compatibility.

Hope this helps.

Aurify answered 30/6, 2009 at 0:13 Comment(0)

boost.serialization just needs the C++ compiler and gives you some syntax sugar like

serialize_obj >> archive;
// ...
unserialize_obj << archive;

for saving and loading. If C++ is the only language you use you should give boost.serialization a serious shot.

I took a fast look at google protocol buffers. From what I see I'd say its not directly comparable to boost.serialization. You have to add a compiler for the .proto files to your toolchain and maintain the .proto files itself. The API doesn't integrate into C++ as boost.serialization does.

boost.serialization does the job its designed for very well: to serialize C++ objects :) OTOH an query-API like google protocol buffers has gives you more flexibility.

Since I only used boost.serialization so far I cannot comment on performance comparison.

Hairpiece answered 30/6, 2009 at 7:25 Comment(0)

Correction to above (guess this is that answer) about Boost Serialization :

It DOES allow supporting data versioning.

If you need compression - use a compressed stream.

Can handle endian swapping between platforms as encoding can be text, binary or XML.

Imray answered 7/6, 2010 at 21:10 Comment(0)

I never implemented anything using boost's library, but I found Google protobuff's to be more thought-out, and the code is much cleaner and easier to read. I would suggest having a look at the various languages you want to use it with and have a read through the code and the documentation and make up your mind.

The one difficulty I had with protobufs was they named a very commonly used function in their generated code GetMessage(), which of course conflicts with the Win32 GetMessage macro.

I would still highly recommend protobufs. They're very useful.

Afro answered 30/6, 2009 at 0:9 Comment(1)

I'll +1 when my limit is restored. Thanks for mentioning the GetMessage hurdle. Something to think about... – Nut 13/7, 2012 at 11:18

I know that this is an older question now, but I thought I'd throw my 2 pence in!

With boost you get the opportunity to I'm write some data validation in your classes; this is good because the data definition and the checks for validity are all in one place.

With GPB the best you can do is to put comments in the .proto file and hope against all hope that whoever is using it reads it, pays attention to it, and implements the validity checks themselves.

Needless to say this is unlikely and unreliable if your relying on someone else at the other end of a network stream to do this with the same vigour as oneself. Plus if the constraints on validity change, multiple code changes need to be planned, coordinated and done.

Thus I consider GPB to be inappropriate for developments where there is little opportunity to regularly meet and talk with all team members.

==EDIT==

The kind of thing I mean is this:

message Foo
{
    int32 bearing = 1;
}

Now who's to say what the valid range of bearing is? We can have

message Foo
{
    int32 bearing = 1;  // Valid between 0 and 359
}

But that depends on someone else reading this and writing code for it. For example, if you edit it and the constraint becomes:

message Foo
{
    int32 bearing = 1;  // Valid between -180 and +180
}

you are completely dependent on everyone who has used this .proto updating their code. That is unreliable and expensive.

At least with Boost serialisation you're distributing a single C++ class, and that can have data validity checks built right into it. If those constraints change, then no one else need do any work other than making sure they're using the same version of the source code as you.

Alternative

There is an alternative: ASN.1. This is ancient, but has some really, really, handy things:

Foo ::= SEQUENCE
{
   bearing INTEGER (0..359)
}

Note the constraint. So whenever anyone consumes this .asn file, generates code, they end up with code that will automatically check that bearing is somewhere between 0 and 359. If you update the .asn file,

Foo ::= SEQUENCE
{
   bearing INTEGER (-180..180)
}

all they need to do is recompile. No other code changes are required.

You can also do:

bearingMin INTEGER ::= 0
bearingMax INTEGER ::= 360

Foo ::= SEQUENCE
{
   bearing INTEGER (bearingMin..<bearingMax)
}

Note the <. And also in most tools the bearingMin and bearingMax can appear as constants in the generated code. That's extremely useful.

Constraints can be quite elaborate:

Garr ::= INTEGER (0..10 | 25..32)

Look at Chapter 13 in this PDF; it's amazing what you can do;

Arrays can be constrained too:

Bar ::= SEQUENCE (SIZE(1..5)) OF Foo
Sna ::= SEQUENCE (SIZE(5)) OF Foo
Fee ::= SEQUENCE 
{
    boo SEQUENCE (SIZE(1..<6)) OF INTEGER (-180<..<180)
}

ASN.1 is old fashioned, but still actively developed, widely used (your mobile phone uses it a lot), and far more flexible than most other serialisation technologies. About the only deficiency that I can see is that there is no decent code generator for Python. If you're using C/C++, C#, Java, ADA then you are well served by a mixture of free (C/C++, ADA) and commercial (C/C++, C#, JAVA) tools.

I especially like the wide choice of binary and text based wireformats. This makes it extremely convenient in some projects. The wireformat list currently includes:

BER (binary)
PER (binary, aligned and unaligned. This is ultra bit efficient. For example, and INTEGER constrained between 0 and 15 will take up only 4 bits on the wire)
OER
DER (another binary)
XML (also XER)
JSON (brand new, tool support is still developing)

plus others.

Note the last two? Yes, you can define data structures in ASN.1, generate code, and emit / consume messages in XML and JSON. Not bad for a technology that started off back in the 1980s.

Versioning is done differently to GPB. You can allow for extensions:

Foo ::= SEQUENCE
{
   bearing INTEGER (-180..180),
   ...
}

This means that at a later date I can add to Foo, and older systems that have this version can still work (but can only access the bearing field).

I rate ASN.1 very highly. It can be a pain to deal with (tools might cost money, the generated code isn't necessarily beautiful, etc). But the constraints are a truly fantastic feature that has saved me a whole ton of heart ache time and time again. Makes developers whinge a lot when the encoders / decoders report that they've generated duff data.

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags