c++ network serialization [closed]

Question

I'm looking for a solution for serializing of c++ packets to a network stream.

I have seen many posts here refering people to:

1. ACE

2. Google Protocol Buffers

3. Boost::Serialization

4. Qt ::QDataStream

My Requirements/ Constraints:

1. The solution must be unaware of LitteEndian/BigEndian. Machine Architecture x86/x64 and platform independant.

2. The foot print (RAM & ROM) of the first 3 solution are too big for my platform,and the fourth is conflicting with the next requirement.

3. The solution won't require a lot of boilerplate code (there will be 200+ packet to be serialized).

Thanks, Koby Meir

Schadenfreude

Hmm, short of RYO this isn't going to happen. Most OS messaging solutions (for example ZeroMq - AFAIK) would use one of the libraries such as ACE or boost::asio. So if you want to avoid, and you don't want to pay, then RYO including low-level socket handling... and good luck. – Discompose

Answer 1

If you find that Google Protocol Buffers are to heavy (I can agree with that because compiled library could take more than 1 MB), you could try the lite version of protobuf which is a few times smaller. It can be enabled in *.proto files by inserting the following line

option optimize_for = LITE_RUNTIME;

But if you need a protobuf solution with minimal overhead I would go with protobuf-c,
a C implementation of protobuf. It will be a little harder to use, but binary code size overhead should be minimal (30-50 KB). I know that this C implementation is used for example by umurmur - a voice server that runs very well on embedded Linux ARM and MIPS routers.

Implausible

Alternative small-footprint protobuf library is nanopb... – Yetta

Answer 2

Another thought: Serialize as text, parsing back on the other side.

We do this a lot (TCP, UDP, serial, other protocols). There is tremendous precedence for this approach, like in robotic control systems, Lab Information Management Systems, and any other place where "hook-up" among many vendors is helpful: Everybody just sends plain text (ASCII or UTF-8), and it's easy to read, easy to debug, easy to reverse-engineer, and easy to repair/hook-into. (If you want it opaque, you can encrypt your payload, like with public/private keys.)

This fits your requirement of agnostic-to-endien-ness/different platform requirements. We've used XML, which works fine and is fairly "standard" with some reference ontology for what you want as (possibly nested) "Key=Value" values, but we tend to prefer an INI-style format, using "named sections" as it gets more complicated. If you "nest" lots of stuff, you might look at JSON-like implementations (it's really easy).

Strong vote for ASCII, IMHO.

Ferro

Answer 3

Wow, ACE, boost serialization ... these frameworks have serialization built in but looking at them for serialization alone is like buying a car because you need CD player. SUN/DEC RPC uses a format called XDR - very well described in Steven's "UNIX Network programming" - essentially it is a 1000 LOC header/c file. that you can use alone. CORBA also uses XDR underneath. Just look for "xdr.h" in Google code - plenty of OSS implementations. If you still need something sophisticated, I would find ASN.1 to be most comprehensive solution which is granted little more complicated than needed for most applications, however ASN.1 compilers generate compact code. It is mostly used by telecom stacks, in cll phones, GSM messaging etc. ASN.1 is used to encode RSA keys.

Nerveless

+1 for the metaphor. ASN.1 is a relevant option, it should have been mentioned, though I'd tend to think about it it still adds a lot to the CD player - at least a huge pile of documentation and installation manuals. – Chiles

Answer 4

I think you'll have to roll your own final solution, but there are a few building blocks you can use:

• Boost::Spirit (Karma to serialize, Qi to deserialize)
  that might still be too big for you, in which case you'll have to roll your own for serialization/deserialization, though you could use the Barton-Nackman idiom to keep the code readable and still use a simply serialize function
• a light-weight communications layer, such as Arachnida (which you could use just for communication, without using its HTTP facilities).
  Arachnida builds on OpenSSL. If that's too heavy for you too, you really should roll your own.

Ignorance/knowledge of little/big endianness would pretty much be in the hands of your serialization code..

Good luck

Philanthropic

Answer 5

I implemented something like this using the same technique microsoft used in MFC. Basically you implement methods in each serializable class to serialize and deserialize whatever you want saved from that class. It's very lightweight and fast.

Instead of the 'archive' class Microsoft used I used a binary stream class. My requirements didn't need endian awareness but it would be trivial to implement if you implement a base class to serialize POD variables.

Baking