Why does CodedInputStream set stream position to end?

Question

I'm using protocol buffers 3 in c#. I'm trying to bounce through a stream to find the start locations of each message, without actually Deserialising the messages. All messages are written to the stream with WriteDelimitedTo.

I then use this code to try to jump from length markers:

_map = new List<int>();
_stream.Seek(0, SeekOrigin.Begin);

var codedStream = new CodedInputStream(_stream);

while (_stream.Position < _stream.Length)
{
    var length = codedStream.ReadInt32();

    _map.Add((int) _stream.Position);

    _stream.Seek(length, SeekOrigin.Current);
}

However, the moment I do codedStream.ReadInt32() the stream position is set to the end, rather than just the next byte after the varint32.

Eustacia

What is _stream? What does it contain? Does it contain more than one integer? Besides, network streams typically can't determine their Length without reading everything first. By checking .Length instead of EOF or the result of Read, you may be reading and discarding everything. That's why most Stream samples check the number of bytes read, not the size. – Bechuana

just a memory stream. and yes, its in a unit test. I write 3 messages and try to jump the length prefixes. but for some reason CodedInputStream doesn't just read the bytes for the varin32, it also seeks the base stream to the end. – Eustacia

Instead of trying to manipulate the underlying stream, use CodedInputStream.IsAtEnd. Touching the underlying stream is a bad idea - you've wrapped the original stream with another, that may buffer or otherwise process its underlying stream. Checking CodedInputStream's code, it seems that is indeed the case. – Bechuana

I don't know about whether it is conveniently available in the specific library you're using, but it is definitely possible to read a varint in a non-buffered way (idea for consuming data without over-reading), then read a length-limited stream or use a length-limited reader. - see here (see also L112-L1142 for the other half); feel free to borrow those two methods if they are useful – Homy

yeah I had your code in the back of my mind as the solution. – Eustacia

Answer 1

This behaviour is due to CodedInputStream buffering the original stream as you can see in the source code. It is probably unsuitable for manually reading and seeking through a stream. An alternative is to use parts of Marc Gravell's source code for reading a varint, available here, and move though the raw stream directly.

Eustacia