does protobuf need a network packet header?

Question

I am using 'protobuf' for C/S network program using TCP.

here is my steps for client:

1, pack data into a 'protobuf'

2, get size in bytes of the pack and construct a length-prefix frame

3, write the frame+pack to socket

and then the server:

1, read Length-prefix frame from socket and get the length N

2, read N bytes from socket and fill the data into protobuf instance

3, get "value"s from protobuf by "key"s

it seems a little complicated I think, is there some kind of auto-generated length-prefix frame for protobuf, then I do not need construct one by myself. or anything else I can do to make code cleaner?

Derzon

Your question is ambiguous / unclear. Are you using TCP? UDP? IP? or some other networking protocol? It is hard to tell what protobuf is. Is it a structure or just a blob of data? Most C/S applications will require some help figuring out the sanity of data to be sent or received. You need to be more specific and show code. – Leading

thanks for your suggestion, I will make it more clear. 'protobuf' refers in particular to a google opensource project. – Derzon

Answer 1

With protobuf, and assuming you are sending multiple messages down the same pipe, then: yes - you will need to have a length as a prefix, otherwise it will want to read to the end of the stream.

protobuf does include some basic RPC stubs, but the RPC implementation they use is not part of the OSS project, so that is not available. There are some standalone protobuf RPC stacks listed in the implementations though.

Personally, I tend to pretend that the sequence of data is part of a repeated sequence - i.e. prefix with "field 1, string" (aka 0a), and the length as "varint" encoded. This means that the entire network stream is a valid protobuf stream. That might just be my OCD kicking in, though.

Some implementations may have features included to help with this. protobuf-net (one of the .NET versions) has SerializeWithLengthPrefix/DeserializeWithLengthPrefix methods, for example, which lets the library do this for you (while offering a range of formats for your choosing).

Coquette

Answer 2

I don't know if this is appropriate for your assignment, but I'd look into one of the 'rpc implementations' already written for the language(s) you want to support.

https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md#rpc-implementations

e.g. I've had good results in Java with Netty's built-in support for sending MessageLite instances of a given type: http://docs.jboss.org/netty/3.2/guide/html/architecture.html#d0e1979

(I'm sure your length header will do the job OK, but frameworks like Netty will add support for things like asynchronous 'duplex' IO, SSL, authentication, etc, with relative ease)

HTH

Zabrine

RPC based on protobuf is a good solution. and I'm still curious about is there any pure protobuf solutions. – Derzon

that's the thing. I think this is outside the scope of protobuf itself. It's just a serialization mechanism, independent from transport or storage. Many transport mechanisms (e.g. one message in a file) don't need to support multiple messages, or provide their own integrity-checking mechanisms. – Zabrine