ZeroMQ + Protocol Buffers

Question

ZeroMQ FAQ page suggest use of Google's protobuf as a way to serialise message content.

Has anyone see a good usage example?

I also need to get the answer to "What is the biggest advantage of serialising messages?" - whether it may be something I can live without and take the advantage of slimmer pipeline.

I quite like the idea of .proto files and the protoc compiler.

Also, it seem that another great tool to throw at the playground would be libev, any comments are welcome :)

Medieval

Just in case if you ask, I am looking into a scalable system for networked music performance (this implies real-time control and doesn't imply audio streaming). Well, I'd like to address a number of issues which OSC protocol doesn't address really. That's said just in case if anyone who reads this is actually familiar with the particular subject, but the question is supposed to be more general. – Medieval

Here are someone's experience using 0MQ + protobuf in 2015, if anyone is interested. – Transient

Answer 1

If you are 100% certain that the programs that are going to communicate over ZMQ will at all times be capable of understanding each other's binary format (eg because they are always distributed together and were all compiled with the same compiler options anyways) I see no benefit to the overhead that's added by serialization.

As soon as the above condition cannot be satisfied (like partner programs running on different host types, programs written in different languages or even partner programs that can evolve independently in time - which may cause incompatibilities in their raw binary structures) serialization becomes quite probably a must.

It seems that nowadays everybody and their brother is creating serialization solutions, which may be an indication that there's no one size fits all solution. This page contains a pretty thorough benchmarking of serialization time, deserialization time and sizes for 27 (!!) different serialization systems. Don't skip the first paragraph of that page, it says "Warning, benchmarks can be misleading". Your application, your data are what counts for you, but the data presented there may help you narrow down the choices you want to study in detail.

Menefee

yeah, it will probably need to run on different architectures, so I'm opting for msgpack :) – Medieval

Interesting Answer. Your comment " I see no benefit to the overhead that's added by serialization." has peaked my interest. How would you intend sending data over the wire without having to pay some form of computational overhead? Weather it's encoding your message to JSON or using some other library such as Protocol buffers there is still a computational cost. Your answer suggests that this can be avoided. How? – Bunsen

@Bunsen the exact conditions that lead to that statement are laid out just before the statement :) : if one is 100% certain that the receiving end and sending end are using exactly the same binary format, serializing doesn't bring any benefits. That's of course a very limiting requirement, but pointing it out seemed useful. – Menefee

@Menefee I have never used binary in this way. I have asked another question on this topic. Would love to hear an answer from you. #53908876 – Bunsen

Answer 2

Here is a sample which send and receive messages through java and in C++:

Serializing in java:

Person person = Person.newBuilder().setName("chand")
    .setEmail("[email protected]").setId(55555).build();
socket.send(person.toByteArray(), 0);

De-serialize in java:

byte[] reply = socket.recv(0);
Person person2 = Person.parseFrom(reply);

Serializing in C++:

Person p = Person();
std::string str;
p.SerializeToString(&str);
int sz = str.length();
zmq::message_t *query = new message_t(sz);
memcpy(query->data (), str.c_str(), sz);
socket->send (*query);

De-serializign in C++

zmq::message_t resultset(100);
socket->recv (&resultset);

Person p = Person();
p.ParseFromArray(resultset.data(), resultset.size());
printf("\n Server : %s", p.name().c_str());

Immutable

Switching from a zmq message to a pointer to a zmq message, as you show, fixed my zmq assertion error that I enountered on send of a protobuf message. – Blackman

Answer 3

I am not sure PUB/SUB in 0mq will work with protobuf, because 0mq expects a string topic at head of the msg.. but protobuf puts a field descriptor first.

actually here is a link with a solution.

http://www.dotkam.com/2011/09/09/zeromq-and-google-protocol-buffers/

cheers

Fremont

the first zmq frame should be the subscription topic when using pubsub. the 2,3,..n frames could be protobuf. – Hallowell

Answer 4

You always need to serialize when communicating. Structures are random access. Communication layers, like ZeroMQ are serial.

You can use the "default serialization" that comes with your language.

For example in C++ a structure with no pointers will have a certain binary layout that can be directly turned into a byte array. This binary layout is, indirectly, your serialization layer, and is both language and compiler specific.

As long as you limit yourself to structures that have no pointers and are using the same compiler and language on both ends of the pipe... feel free to avoid a library that does additional serialization on top of the default layout provided.

Peppie