Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?
Asked Answered
M

11

71

I'm trying to read / write multiple Protocol Buffers messages from files, in both C++ and Java. Google suggests writing length prefixes before the messages, but there's no way to do that by default (that I could see).

However, the Java API in version 2.1.0 received a set of "Delimited" I/O functions which apparently do that job:

parseDelimitedFrom
mergeDelimitedFrom
writeDelimitedTo

Are there C++ equivalents? And if not, what's the wire format for the size prefixes the Java API attaches, so I can parse those messages in C++?


Update:

These now exist in google/protobuf/util/delimited_message_util.h as of v3.3.0.

Meretricious answered 26/2, 2010 at 9:58 Comment(4)
I don't know, but it's open-source so you can find out from the source.Alephnull
Yeah, that's what I ended up doing. :) See my answer below.Meretricious
As of v3.3.0 google::protobuf::util offers the delimited message methods for MessageLite.Donothingism
@KenjiNoguchi Thanks for the tip! I updated the question to include that.Meretricious
C
83

I'm a bit late to the party here, but the below implementations include some optimizations missing from the other answers and will not fail after 64MB of input (though it still enforces the 64MB limit on each individual message, just not on the whole stream).

(I am the author of the C++ and Java protobuf libraries, but I no longer work for Google. Sorry that this code never made it into the official lib. This is what it would look like if it had.)

bool writeDelimitedTo(
    const google::protobuf::MessageLite& message,
    google::protobuf::io::ZeroCopyOutputStream* rawOutput) {
  // We create a new coded stream for each message.  Don't worry, this is fast.
  google::protobuf::io::CodedOutputStream output(rawOutput);

  // Write the size.
  const int size = message.ByteSize();
  output.WriteVarint32(size);

  uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
  if (buffer != NULL) {
    // Optimization:  The message fits in one buffer, so use the faster
    // direct-to-array serialization path.
    message.SerializeWithCachedSizesToArray(buffer);
  } else {
    // Slightly-slower path when the message is multiple buffers.
    message.SerializeWithCachedSizes(&output);
    if (output.HadError()) return false;
  }

  return true;
}

bool readDelimitedFrom(
    google::protobuf::io::ZeroCopyInputStream* rawInput,
    google::protobuf::MessageLite* message) {
  // We create a new coded stream for each message.  Don't worry, this is fast,
  // and it makes sure the 64MB total size limit is imposed per-message rather
  // than on the whole stream.  (See the CodedInputStream interface for more
  // info on this limit.)
  google::protobuf::io::CodedInputStream input(rawInput);

  // Read the size.
  uint32_t size;
  if (!input.ReadVarint32(&size)) return false;

  // Tell the stream not to read beyond that size.
  google::protobuf::io::CodedInputStream::Limit limit =
      input.PushLimit(size);

  // Parse the message.
  if (!message->MergeFromCodedStream(&input)) return false;
  if (!input.ConsumedEntireMessage()) return false;

  // Release the limit.
  input.PopLimit(limit);

  return true;
}
Confection answered 8/4, 2014 at 3:49 Comment(24)
Hey, thanks for the answer Kenton! I'll switch accepted to this one over my own. Although I suspect the best answer at this point is to use Cap'n Proto instead? :)Meretricious
Also - why not get this merged in to the official protobuf lib on code.google?Meretricious
As an official patch it would need some more work, like unit tests, maybe a better error recovery story, etc., and between Cap'n Proto and Sandstorm.io I just don't have time. :/ If someone wants to claim this as their own and push it upstream feel free. The two functions should probably become methods on MessageLite. You should probably discuss with the current maintainers before doing any work as they might have their own plans.Confection
what about creating an OstreamOutputStream every time (wrapping a std::ostream arg), is that fast too?Prosser
@Prosser - I think OstreamOutputStream is an instance of CopyingOutputStream which allocates a buffer, therefore it is slow to construct and destruct. Also, on destruction, any unread data currently in the buffer is simply discarded (not rewound), so you may be left at an unspecified offset in the stream.Confection
I just noticed that the second part of my previous comment is wrong. The concern about unread data in the buffer only applies to input streams (e.g. IstreamInputStream), not output streams. The first part of the comment still applies, though.Confection
A downside to using a varint header is that it's tricky to specify to an asynchronous API (like ASIO) to notify you when a whole header has been read. Using fixed sized ints is trivial: you just ask to wait until 4 bytes have been received (with asio::transfer_at_least). For varints you'd want to optimise for the common case that a whole header is read at once, while avoiding quadratic behaviour if someone sends an infinite stream of bytes with the high bit set. Also, to me, all this logic feels a bit too high level for socket reading code.Ashlieashlin
Unrelated to my last comment: If someone did integrate this into the main protobuf project, it would be great if support were also included in protoc's --encode/decode switches (with multiple messages) for reading and writing binary protobuf files.Ashlieashlin
Oh yeah, I should probably mention I submitted a pull request to protobuf to add these functions three months ago... hasn't been accepted yet, though: github.com/google/protobuf/pull/710Confection
@KentonVarda What would be the python equivalent of those functions?Schorl
@fireboot Sorry, I didn't write the Python library so I'm not as familiar with it. I'd have to dig around for a while to figure it out, and unfortunately I don't have time. :/ I could probably verify code produced by someone else, though.Confection
@KentonVarda I will give it a go in the next days and post it hereSchorl
@KentonVarda I just posted my version in this thread. Comments welcome :)Schorl
@kentonVarda, i'm looking for a way to write for example multiple users using a user message, then being able to get one specific user using the byte position of that user, is this the solution I need?Jacobson
@JimOldfield - totally agreed on the varints. If you control both sides of the connection, doing a 4 byte length greatly simplifies things.Ozalid
@KentonVarda this code looks great! But I can't make it work. I'm passing a new ::google::protobuf::io::OstreamOutputStream(&file) as rawInput, with a message 9458 bytes long, it generates a file 8192 bytes long... Of course, deserialization fails, message is truncated. Some thoughts?Ezequiel
@AlejandroSilvestri It sounds like some buffers haven't been flushed. I believe both OstreamOutputStream and std::ostream buffer data. They'll flush that data when you destroy the objects. I think they might also have explicit flush methods you can call, if you don't want to destroy the objects yet.Confection
Confirmed. As @KentonVarda said, I had to destroy OstreamOutputStream before closing the ofstream file.Ezequiel
Works perfectely ! :) I used Parse/SerializePartialFromZeroCopyStream, it seemed like the writing was OK (large file), but I could parse only the last message. Is it a problem of file cursor ? I don't get the differences between Parse/Serialize.. from read/writeDelimited... except the two last write the size of the messageDonnie
Had to read this twice bc I implied that delimited means you would write some delimiter after the message. Instead you just write this size first...Source
FYI, the PR listed above has been merged and the functions in this answer are now available in delimited_message_util.h.Ribeiro
Beware that in recent version of protobuf, google::protobuf::io::CodedInputStream input(rawInput); will immediately buffer/read lots of bytes, so don't rely on your IstreamInputStream stream object to know the actual offset of the current read message. That'is after reading first message, stream.tell() maybe at the end of the stream and not at the end of the first message in the stream!Dreamadreamer
@Dreamadreamer That has always been true. CodedInputStream asks the underlying ZeroCopyInputStream for a whole buffer of data immediately. CodedInputStream's destructor returns whatever portion of the buffer wasn't used. So if the CodedInputStream is still live, then the underlying ZeroCopyInputStream will be in an indeterminate state. That is how it worked when I wrote it in 2006...Confection
You are correct @KentonVarda . I confirmed that was the same in early version, it was a change of how we use it that caused the issue for us. Thank you for the correction!Dreamadreamer
M
17

Okay, so I haven't been able to find top-level C++ functions implementing what I need, but some spelunking through the Java API reference turned up the following, inside the MessageLite interface:

void writeDelimitedTo(OutputStream output)
/*  Like writeTo(OutputStream), but writes the size of 
    the message as a varint before writing the data.   */

So the Java size prefix is a (Protocol Buffers) varint!

Armed with that information, I went digging through the C++ API and found the CodedStream header, which has these:

bool CodedInputStream::ReadVarint32(uint32 * value)
void CodedOutputStream::WriteVarint32(uint32 value)

Using those, I should be able to roll my own C++ functions that do the job.

They should really add this to the main Message API though; it's missing functionality considering Java has it, and so does Marc Gravell's excellent protobuf-net C# port (via SerializeWithLengthPrefix and DeserializeWithLengthPrefix).

Meretricious answered 26/2, 2010 at 12:53 Comment(1)
Yes. This is the way I solved this problem. I added another answer with some sample pseudo code for writing a message.Puzzlement
P
12

I solved the same problem using CodedOutputStream/ArrayOutputStream to write the message (with the size) and CodedInputStream/ArrayInputStream to read the message (with the size).

For example, the following pseudo-code writes the message size following by the message:

const unsigned bufLength = 256;
unsigned char buffer[bufLength];
Message protoMessage;

google::protobuf::io::ArrayOutputStream arrayOutput(buffer, bufLength);
google::protobuf::io::CodedOutputStream codedOutput(&arrayOutput);

codedOutput.WriteLittleEndian32(protoMessage.ByteSize());
protoMessage.SerializeToCodedStream(&codedOutput);

When writing you should also check that your buffer is large enough to fit the message (including the size). And when reading, you should check that your buffer contains a whole message (including the size).

It definitely would be handy if they added convenience methods to C++ API similar to those provided by the Java API.

Puzzlement answered 26/2, 2010 at 13:19 Comment(1)
I'll be using an underlying OstreamOutputStream, so the length-checking won't be necessary, but thanks for the answer. :) In your case, I'd probably go with setting the bufLength to protoMessage.ByteSize() plus some extra for the size prefix.Meretricious
T
8

IsteamInputStream is very fragile to eofs and other errors that easily occurs when used together with std::istream. After this the protobuf streams are permamently damaged and any already used buffer data is destroyed. There are proper support for reading from traditional streams in protobuf.

Implement google::protobuf::io::CopyingInputStream and use that together with CopyingInputStreamAdapter. Do the same for the output variants.

In practice a parsing call ends up in google::protobuf::io::CopyingInputStream::Read(void* buffer, int size) where a buffer is given. The only thing left to do is read into it somehow.

Here's an example for use with Asio synchronized streams (SyncReadStream/SyncWriteStream):

#include <google/protobuf/io/zero_copy_stream_impl_lite.h>

using namespace google::protobuf::io;


template <typename SyncReadStream>
class AsioInputStream : public CopyingInputStream {
    public:
        AsioInputStream(SyncReadStream& sock);
        int Read(void* buffer, int size);
    private:
        SyncReadStream& m_Socket;
};


template <typename SyncReadStream>
AsioInputStream<SyncReadStream>::AsioInputStream(SyncReadStream& sock) :
    m_Socket(sock) {}


template <typename SyncReadStream>
int
AsioInputStream<SyncReadStream>::Read(void* buffer, int size)
{
    std::size_t bytes_read;
    boost::system::error_code ec;
    bytes_read = m_Socket.read_some(boost::asio::buffer(buffer, size), ec);

    if(!ec) {
        return bytes_read;
    } else if (ec == boost::asio::error::eof) {
        return 0;
    } else {
        return -1;
    }
}


template <typename SyncWriteStream>
class AsioOutputStream : public CopyingOutputStream {
    public:
        AsioOutputStream(SyncWriteStream& sock);
        bool Write(const void* buffer, int size);
    private:
        SyncWriteStream& m_Socket;
};


template <typename SyncWriteStream>
AsioOutputStream<SyncWriteStream>::AsioOutputStream(SyncWriteStream& sock) :
    m_Socket(sock) {}


template <typename SyncWriteStream>
bool
AsioOutputStream<SyncWriteStream>::Write(const void* buffer, int size)
{   
    boost::system::error_code ec;
    m_Socket.write_some(boost::asio::buffer(buffer, size), ec);
    return !ec;
}

Usage:

AsioInputStream<boost::asio::ip::tcp::socket> ais(m_Socket); // Where m_Socket is a instance of boost::asio::ip::tcp::socket
CopyingInputStreamAdaptor cis_adp(&ais);
CodedInputStream cis(&cis_adp);

Message protoMessage;
uint32_t msg_size;

/* Read message size */
if(!cis.ReadVarint32(&msg_size)) {
    // Handle error
 }

/* Make sure not to read beyond limit of message */
CodedInputStream::Limit msg_limit = cis.PushLimit(msg_size);
if(!msg.ParseFromCodedStream(&cis)) {
    // Handle error
}

/* Remove limit */
cis.PopLimit(msg_limit);
Thierry answered 26/2, 2010 at 9:58 Comment(1)
This was a huge help. I had tried doing protobuf over sockets by using the asio istream/ostream interface and wrapping them in IStreamInputStream/OStreamOutputStream, and couldn't get it working. Thanks for posting this. With it and Kenton's functions, you can fairly easily build a client/server to talk protobuf in c++ using asio.Ozalid
O
7

Here you go:

#include <google/protobuf/io/zero_copy_stream_impl.h>
#include <google/protobuf/io/coded_stream.h>

using namespace google::protobuf::io;

class FASWriter 
{
    std::ofstream mFs;
    OstreamOutputStream *_OstreamOutputStream;
    CodedOutputStream *_CodedOutputStream;
public:
    FASWriter(const std::string &file) : mFs(file,std::ios::out | std::ios::binary)
    {
        assert(mFs.good());

        _OstreamOutputStream = new OstreamOutputStream(&mFs);
        _CodedOutputStream = new CodedOutputStream(_OstreamOutputStream);
    }

    inline void operator()(const ::google::protobuf::Message &msg)
    {
        _CodedOutputStream->WriteVarint32(msg.ByteSize());

        if ( !msg.SerializeToCodedStream(_CodedOutputStream) )
            std::cout << "SerializeToCodedStream error " << std::endl;
    }

    ~FASWriter()
    {
        delete _CodedOutputStream;
        delete _OstreamOutputStream;
        mFs.close();
    }
};

class FASReader
{
    std::ifstream mFs;

    IstreamInputStream *_IstreamInputStream;
    CodedInputStream *_CodedInputStream;
public:
    FASReader(const std::string &file), mFs(file,std::ios::in | std::ios::binary)
    {
        assert(mFs.good());

        _IstreamInputStream = new IstreamInputStream(&mFs);
        _CodedInputStream = new CodedInputStream(_IstreamInputStream);      
    }

    template<class T>
    bool ReadNext()
    {
        T msg;
        unsigned __int32 size;

        bool ret;
        if ( ret = _CodedInputStream->ReadVarint32(&size) )
        {   
            CodedInputStream::Limit msgLimit = _CodedInputStream->PushLimit(size);
            if ( ret = msg.ParseFromCodedStream(_CodedInputStream) )
            {
                _CodedInputStream->PopLimit(msgLimit);      
                std::cout << mFeed << " FASReader ReadNext: " << msg.DebugString() << std::endl;
            }
        }

        return ret;
    }

    ~FASReader()
    {
        delete _CodedInputStream;
        delete _IstreamInputStream;
        mFs.close();
    }
};
Orly answered 2/10, 2012 at 6:6 Comment(0)
S
7

I ran into the same issue in both C++ and Python.

For the C++ version, I used a mix of the code Kenton Varda posted on this thread and the code from the pull request he sent to the protobuf team (because the version posted here doesn't handle EOF while the one he sent to github does).

#include <google/protobuf/message_lite.h>
#include <google/protobuf/io/zero_copy_stream.h>
#include <google/protobuf/io/coded_stream.h>


bool writeDelimitedTo(const google::protobuf::MessageLite& message,
    google::protobuf::io::ZeroCopyOutputStream* rawOutput)
{
    // We create a new coded stream for each message.  Don't worry, this is fast.
    google::protobuf::io::CodedOutputStream output(rawOutput);

    // Write the size.
    const int size = message.ByteSize();
    output.WriteVarint32(size);

    uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
    if (buffer != NULL)
    {
        // Optimization:  The message fits in one buffer, so use the faster
        // direct-to-array serialization path.
        message.SerializeWithCachedSizesToArray(buffer);
    }

    else
    {
        // Slightly-slower path when the message is multiple buffers.
        message.SerializeWithCachedSizes(&output);
        if (output.HadError())
            return false;
    }

    return true;
}

bool readDelimitedFrom(google::protobuf::io::ZeroCopyInputStream* rawInput, google::protobuf::MessageLite* message, bool* clean_eof)
{
    // We create a new coded stream for each message.  Don't worry, this is fast,
    // and it makes sure the 64MB total size limit is imposed per-message rather
    // than on the whole stream.  (See the CodedInputStream interface for more
    // info on this limit.)
    google::protobuf::io::CodedInputStream input(rawInput);
    const int start = input.CurrentPosition();
    if (clean_eof)
        *clean_eof = false;


    // Read the size.
    uint32_t size;
    if (!input.ReadVarint32(&size))
    {
        if (clean_eof)
            *clean_eof = input.CurrentPosition() == start;
        return false;
    }
    // Tell the stream not to read beyond that size.
    google::protobuf::io::CodedInputStream::Limit limit = input.PushLimit(size);

    // Parse the message.
    if (!message->MergeFromCodedStream(&input)) return false;
    if (!input.ConsumedEntireMessage()) return false;

    // Release the limit.
    input.PopLimit(limit);

    return true;
}

And here is my python2 implementation:

from google.protobuf.internal import encoder
from google.protobuf.internal import decoder

#I had to implement this because the tools in google.protobuf.internal.decoder
#read from a buffer, not from a file-like objcet
def readRawVarint32(stream):
    mask = 0x80 # (1 << 7)
    raw_varint32 = []
    while 1:
        b = stream.read(1)
        #eof
        if b == "":
            break
        raw_varint32.append(b)
        if not (ord(b) & mask):
            #we found a byte starting with a 0, which means it's the last byte of this varint
            break
    return raw_varint32

def writeDelimitedTo(message, stream):
    message_str = message.SerializeToString()
    delimiter = encoder._VarintBytes(len(message_str))
    stream.write(delimiter + message_str)

def readDelimitedFrom(MessageType, stream):
    raw_varint32 = readRawVarint32(stream)
    message = None

    if raw_varint32:
        size, _ = decoder._DecodeVarint32(raw_varint32, 0)

        data = stream.read(size)
        if len(data) < size:
            raise Exception("Unexpected end of file")

        message = MessageType()
        message.ParseFromString(data)

    return message

#In place version that takes an already built protobuf object
#In my tests, this is around 20% faster than the other version 
#of readDelimitedFrom()
def readDelimitedFrom_inplace(message, stream):
    raw_varint32 = readRawVarint32(stream)

    if raw_varint32:
        size, _ = decoder._DecodeVarint32(raw_varint32, 0)

        data = stream.read(size)
        if len(data) < size:
            raise Exception("Unexpected end of file")

        message.ParseFromString(data)

        return message
    else:
        return None

It might not be the best looking code and I'm sure it can be refactored a fair bit, but at least that should show you one way to do it.

Now the big problem: It's SLOW.

Even when using the C++ implementation of python-protobuf, it's one order of magnitude slower than in pure C++. I have a benchmark where I read 10M protobuf messages of ~30 bytes each from a file. It takes ~0.9s in C++, and 35s in python.

One way to make it a bit faster would be to re-implement the varint decoder to make it read from a file and decode in one go, instead of reading from a file and then decoding as this code currently does. (profiling shows that a significant amount of time is spent in the varint encoder/decoder). But needless to say that alone is not enough to close the gap between the python version and the C++ version.

Any idea to make it faster is very welcome :)

Schorl answered 31/12, 2015 at 1:13 Comment(4)
There is a general question why there are different implementations for encoding/decoding in Java/Python/C++. I do not understand why there is no base implementation in C++, which simply gets called in Java/Python...Cladoceran
Your python code does not seem to work when using Python3. You would need to read bytes instead of strings for decoder to work.Cladoceran
Yes, this code was written for python 2, but it should be rather easy to adapt it and make it work for python 3. I've edited my post to indicate that this code targets python 2.Schorl
Can you please confirm that stream is of type StringIO in pythonCrake
L
6

Just for completeness, I post here an up-to-date version that works with the master version of protobuf and Python3

For the C++ version it is sufficient to use the utils in delimited_message_utils.h, here a MWE

#include <google/protobuf/io/zero_copy_stream_impl.h>
#include <google/protobuf/util/delimited_message_util.h>

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

template <typename T>
bool writeManyToFile(std::deque<T> messages, std::string filename) {
    int outfd = open(filename.c_str(), O_WRONLY | O_CREAT | O_TRUNC);
    google::protobuf::io::FileOutputStream fout(outfd);

    bool success;
    for (auto msg: messages) {
        success = google::protobuf::util::SerializeDelimitedToZeroCopyStream(
            msg, &fout);
        if (! success) {
            std::cout << "Writing Failed" << std::endl;
            break;
        }
    }
    fout.Close();
    close(outfd);
    return success;
}

template <typename T>
std::deque<T> readManyFromFile(std::string filename) {
    int infd = open(filename.c_str(), O_RDONLY);

    google::protobuf::io::FileInputStream fin(infd);
    bool keep = true;
    bool clean_eof = true;
    std::deque<T> out;

    while (keep) {
        T msg;
        keep = google::protobuf::util::ParseDelimitedFromZeroCopyStream(
            &msg, &fin, nullptr);
        if (keep)
            out.push_back(msg);
    }
    fin.Close();
    close(infd);
    return out;
}

For the Python3 version, building on @fireboot 's answer, the only thing thing that needed modification is the decoding of raw_varint32

def getSize(raw_varint32):
    result = 0
    shift = 0
    b = six.indexbytes(raw_varint32, 0)
    result |= ((ord(b) & 0x7f) << shift)
    return result

def readDelimitedFrom(MessageType, stream):
    raw_varint32 = readRawVarint32(stream)
    message = None

    if raw_varint32:
        size = getSize(raw_varint32)

        data = stream.read(size)
        if len(data) < size:
            raise Exception("Unexpected end of file")

        message = MessageType()
        message.ParseFromString(data)

    return message
Labannah answered 27/11, 2019 at 15:31 Comment(1)
This solution deserves to be much higher rated. Older answers don't reflect how the protobuf C++ lib has evolved to resolve what was obviously quite a shortcoming.Approver
B
3

Was also looking for a solution for this. Here's the core of our solution, assuming some java code wrote many MyRecord messages with writeDelimitedTo into a file. Open the file and loop, doing:

if(someCodedInputStream->ReadVarint32(&bytes)) {
  CodedInputStream::Limit msgLimit = someCodedInputStream->PushLimit(bytes);
  if(myRecord->ParseFromCodedStream(someCodedInputStream)) {
    //do your stuff with the parsed MyRecord instance
  } else {
    //handle parse error
  }
  someCodedInputStream->PopLimit(msgLimit);
} else {
  //maybe end of file
}

Hope it helps.

Bushtit answered 30/6, 2011 at 9:43 Comment(0)
S
0

Working with an objective-c version of protocol-buffers, I ran into this exact issue. On sending from the iOS client to a Java based server that uses parseDelimitedFrom, which expects the length as the first byte, I needed to call writeRawByte to the CodedOutputStream first. Posting here to hopegully help others that run into this issue. While working through this issue, one would think that Google proto-bufs would come with a simply flag which does this for you...

    Request* request = [rBuild build];

    [self sendMessage:request];
} 


- (void) sendMessage:(Request *) request {

    //** get length
    NSData* n = [request data];
    uint8_t len = [n length];

    PBCodedOutputStream* os = [PBCodedOutputStream streamWithOutputStream:outputStream];
    //** prepend it to message, such that Request.parseDelimitedFrom(in) can parse it properly
    [os writeRawByte:len];
    [request writeToCodedOutputStream:os];
    [os flush];
}
Scarlettscarp answered 13/12, 2013 at 21:24 Comment(0)
E
0

Since I'm not allowed to write this as a comment to Kenton Varda's answer above; I believe there is a bug in the code he posted (as well as in other answers which have been provided). The following code:

...
google::protobuf::io::CodedInputStream input(rawInput);

// Read the size.
uint32_t size;
if (!input.ReadVarint32(&size)) return false;

// Tell the stream not to read beyond that size.
google::protobuf::io::CodedInputStream::Limit limit =
    input.PushLimit(size);
...

sets an incorrect limit because it does not take into account the size of the varint32 which has already been read from input. This can result in data loss/corruption as additional bytes are read from the stream which may be part of the next message. The usual way of handling this correctly is to delete the CodedInputStream used to read the size and create a new one for reading the payload:

...
uint32_t size;
{
  google::protobuf::io::CodedInputStream input(rawInput);

  // Read the size.
  if (!input.ReadVarint32(&size)) return false;
}

google::protobuf::io::CodedInputStream input(rawInput);

// Tell the stream not to read beyond that size.
google::protobuf::io::CodedInputStream::Limit limit =
    input.PushLimit(size);
...
Ecliptic answered 5/4, 2016 at 1:59 Comment(4)
That would only be true if the size prefix included its own size, which it doesn't. If you do this you'll end up not reading the whole message.Meretricious
It is precisely because the size prefix does not include its own size that this is a problem.Ecliptic
The size prefix contains exactly the size of the message, which follows it. The code then proceeds to read that many bytes, which contain the entire message. Where's the problem?Meretricious
Both the original code and the version I posted work fine and it turns out this wasn't my problem after all. My issue was CodedInputStream unexpectedly consuming all the data from the source buffer even though a limit had been set. I was trying to determine how much data was left over and CodedInputStream makes that very difficult. While in C#, this question helped me to figure it out: #33734413Ecliptic
V
-7

You can use getline for reading a string from a stream, using the specified delimiter:

istream& getline ( istream& is, string& str, char delim );

(defined in the header)

Vipul answered 26/2, 2010 at 10:20 Comment(1)
Not the same thing; protocol buffers is a binary format, the "Delimited" functions actually just prepend a size. I'd need to know the format of the size prefix.Meretricious

© 2022 - 2024 — McMap. All rights reserved.