Google Protocol Buffers - Missing required fields even though all the fields are apparently present

Question

I'm trying to send a protocol buffer message over TCP, but on the receiving side I'm getting a "Missing required fields" error when trying to parse, even though apparently all the fields are there. I'm sending a 4 byte header before the message that contains the length of the message.

Here is the message definition:

message ReplayRequest {
  required string channel = 1;
  required uint32 start = 2;
  required uint32 end = 3;
}

On the client side I'm encoding the header and serializing the message into a vector.

ReplayRequest req;
req.set_channel( "channel" )
req.set_start( 1 );
req.set_end( 5 );
int byte_size = req.ByteSize();
std::vector<uint8_t> write_buffer( HEADER_SIZE + byte_size );
encode_header( ... );
req.SerializeToArray( &write_buffer[HEADER_SIZE], byte_size );

This is a hex print of the resulting buffer, where the first 4 bytes are the encoded message length (13 bytes).

00 00 00 0d 0a 07 63 68 61 6e 6e 65 6c 10 01 18 05

On the server side, I receive the header, decode it and then receive N bytes where N is the message size reported in the header. The buffer in the server with the header removed is:

0a 07 63 68 61 6e 6e 65 6c 10 01 18 05

Which is exactly the same as the one encoded client side minus the header, but when I try to ParseFromArray this buffer I get an error:

libprotobuf ERROR c:\umdf_runtime\protobuf-2.4.1\src\google\protobuf\message_lit
e.cc:123] Can't parse message of type "ReplayRequest" because it is missing 
required fields: channel, start, end

While debugging I noticed that the point where decoding fails is on this part of the protobuf generated code:

bool ReplayRequest::IsInitialized() const {
  if ((_has_bits_[0] & 0x00000007) != 0x00000007) return false;

  return true;
}

has_bits_ is being read as zero on the server side for some reason but I can't figure out why.

Any ideas?

I'm using boost::asio for the network part if it matters.

Update

As requested I'm posting the code that calls parseFromArray.

request_.ParseFromArray( &data_buffer_, data_buffer_.size() );

request_ is a ReplayRequest member variable, up until this call nothing is done to it.

data_buffer_ is a vector<uint8_t> where the TCP data is received into.

I confirmed that it is correctly sized at 13 bytes and this is its hex dump, which is the same that I get when I dump the buffer client side after serializing.

0a 07 63 68 61 6e 6e 65 6c 10 01 18 05

Update 2

I am able to parse the buffer into another instance of ReplayRequest on the client side, i.e.:

...snip...
req.SerializeToArray( &write_buffer[HEADER_SIZE], byte_size );
ReplayRequest test;
test.ParseFromArray( &write_buffer[HEADER_SIZE], byte_size );

test is successfully populated with the correct fields.

Jackie

If the server says that it doesn't have any of the required fields, then I'm wondering if you're extracting and parsing the message incorrectly on the server. Could you please post the code that calls ParseFromArray? – Laciniate

I notice your four-byte encoded message size header seems to be big-endian... are you decoding it correctly as big-endian on the receiver side? I guess you must be decoding it correctly because you get the correct N. – Darton

@James The header is decoded correctly, you can see that the server buffer I printed is exactly 13 bytes, as encoded. – Jackie

are you sure you're not using different 'compiled' versions of your protos between the different programs? Just to be sure, I'd recompile the .proto file to the necessary c++ files and then copy those directly into both projects and re-build. Also as a test, are you able to deserialize the message in your first program (using a different message instance, not using the same instance in which you serialized)? – Inger

@Inger The compiled message is in a static library which is linked from both the client and the server, so it should be the same one. I can deserialize into a different instance, please check Update 2 to the question. – Jackie

@josh-kelley please check the update in the question. – Jackie

Answer 1

The issue is that you're passing in a pointer to the vector and not a pointer to the vector's data.

instead of
request_.ParseFromArray( &data_buffer_, data_buffer_.size() );

try
request_.ParseFromArray( &data_buffer_[0], data_buffer_.size() );

Inger

That's it. I received into a single buffer on the server before, so &data_buffer_[HEADER_SIZE] was working fine. After I split the buffers I removed the accessor. Thank you. – Jackie

Answer 2

Another solution if the required field is missing but not needed for you and if you cannot change that field to optional in the protofile, you could use ParsePartialFromArray instead of ParseFromArray.

See protobuf documentation: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message_lite

Outrageous