Sending Protobuf Messages with boost::asio

Question

I'm trying to hack a client together in C++ using Google's Protocol Buffers and boost::asio.

My problem is that I don't know how I can feed the protobuf message to asio. What I have is this:

// set up *sock - works
PlayerInfo info;
info.set_name(name);
// other stuff

Now I know that the following is wrong, but I'll post it anyways:

size_t request_length = info.ByteSize();
boost::asio::write(*sock, boost::asio::buffer(info, request_length));

I got as far as that I know that I have to pack my message differently into the buffer - but how?

Generally speaking, I'm having a hard time figuring out how boost::asio works. There are some tutorials, but they normally just cover sending standard data formats such as ints, which works out-of-the-box. I figured that my problem is serialization, but on the other hand I learned that protobuf should do this for me... and now I'm confused ;)

Thanks for your help!

--> Daniel Gehriger provided the solution, thanks a lot!

Obsess

Answer 1

I don't know much about Google's Protocol buffer, but try the following:

PlayerInfo info;
info.set_name(name);
// ...

boost::asio::streambuf b;
std::ostream os(&b);
info.SerializeToOstream(&os);

boost::asio::write(*sock, b);

Lucey

You will need to add some more information to the stream so that you can reliably read the PlayerInfo object on the other end. Protobuf does not directly embed the object type in the stream. I don't think that it contains the length either, but I would have to check the documentation. BTW, ASIO doesn't know about objects; it only deals with bytes. Keep that in mind when you read the examples and documentation. – Raji

Dan: Yup, I figured I have to add the size beforehand so the server is able to read it... Thanks! – Obsess

@adi64: great! Maybe you would like to mention that in your question, in case it's useful to someone else. – Lucey

@Obsess - thanks for the credits! But I wanted to say that you mention that one also has to send the size beforehand. That part of the code could be of interest. – Lucey

@Daniel Gehriger: ah now I get what you meant ;) Adding the size in beforehand was only important in my specific case, because the receiving server (which I did not write) expects 4 size bytes to be sent before the actual data package. So you can perfectly send your protobuf Message without any size options if you configure your server to rely on boost::asio's packet info. – Obsess

Size prefixed packets is what everyone first tries in network communications, and it works when the world is clean. But note that it doesn't get you out of trouble if you need to resync the stream (i.e., you're not sure where you are). This is why robust protocols often depend on delimiters despite the bother of escaping. – Alver

@Alver this. I've spent near a decade explaining that to devs and each next one would claim "This is on wire. How it can be broken?" They come out of education believing that you have a guarantee to get perfect transmission without any noise. – Haubergeon

Answer 2

I've just started using Google Protocol Buffers (protobuf) and also had problems sending (and receiving) messages over a computer network.

In contrast to the Java API, the C++ API does not have a writeDelimitedTo method to send a protobuf message with a delimiter. Without a delimiter we also have to send the size of the message, to be able to de-serialize it at the receive endpoint.

The C++ API offers the class ::google::protobuf::io::CodedOutputStream, defined in the header file google/protobuf/io/coded_stream.h.

The following source code demonstrates how-to send a delimited protobuf message via Boost.Asio over the wire. The example uses UDP. Since I haven't found a working example on the WWW, I share it here.

#include "boost/asio.hpp"
#include "google/protobuf/io/coded_stream.h"
#include "google/protobuf/io/zero_copy_stream_impl.h"

using ::boost::asio::ip::udp;

int main() {
  PlayerInfo message;
  message.set_name("Player 1");
  // ...

  const boost::asio::ip::address_v4 kIpAddress = boost::asio::ip::address_v4::loopback();
  const unsigned short kPortNumber = 65535;

  try {
    boost::asio::io_service io_service;
    udp::socket socket(io_service, boost::asio::ip::udp::v4());

    udp::endpoint endpoint(kIpAddress, kPortNumber);
    boost::system::error_code error;

    boost::asio::streambuf stream_buffer;
    std::ostream output_stream(&stream_buffer);

    {
      ::google::protobuf::io::OstreamOutputStream raw_output_stream(&output_stream);
      ::google::protobuf::io::CodedOutputStream coded_output_stream(&raw_output_stream);
      coded_output_stream.WriteVarint32(message.ByteSize());

      message.SerializeToCodedStream(&coded_output_stream);
      // IMPORTANT: In order to flush a CodedOutputStream it has to be deleted,
      // otherwise a 0 bytes package is send over the wire.
    }
  }

  size_t len = socket.send_to(stream_buffer.data(), endpoint, 0, error);

  if (error && error != boost::asio::error::message_size) {
    throw boost::system::system_error(error);
  }

  std::cout << "Sent " << len << " bytes data to " << kIpAddress.to_string() << "." << std::endl;
} catch (const std::exception& ex) {
  std::cerr << ex.what() << std::endl;
}

While writing this article, I've also discovered the following two questions:

• Google Protocol Buffers: parseDelimitedFrom and writeDelimitedTo for C++
• boost::asio::streambuf empty?

Both are related to this question and also contain (partial) answers. I hope my answer may be useful anyway.

Receptive

Answer 3

• Using asio::streambuf to storage data(also can include custom header)
• Then using boost::smart_ptr::local_shared_ptr() for single thread or std::shared_ptr to delegate the owner of data.

Then send it in any time, for example:

boost::local_shared_ptr<asio::streambuf> wbuf;
asio::async_write(sock, *wbuf, [&, wbuf](const asio::error_code &ec, std::size_t len){});

Liminal