How do I read exactly n bytes from a stream?

Question

This is a little more tricky than I first imagined. I'm trying to read n bytes from a stream.

The MSDN claims that Read does not have to return n bytes, it just must return at least 1 and up to n bytes, with 0 bytes being the special case of reaching the end of the stream.

Typically, I'm using something like

var buf = new byte[size];
var count = stream.Read (buf, 0, size);

if (count != size) {
    buf = buf.Take (count).ToArray ();
}

yield return buf;

I'm hoping for exactly size bytes but by spec FileStream would be allowed to return a large number of 1-byte chunks as well. This must be avoided.

One way to solve this would be to have 2 buffers, one for reading and one for collecting the chunks until we got the requested number of bytes. That's a little cumbersome though.

I also had a look at BinaryReader but its spec also does not clearly state that n bytes will be returned for sure.

To clarify: Of course, upon the end of the stream the returned number of bytes may be less than size - that's not a problem. I'm only talking about not receiving n bytes even though they are available in the stream.

Cryostat

the BinaryReader.ReadBytes(int) returns the number of bytes requested; if the stream ends earlier, it will return what it read until that point (so less than requested). – Pessimism

@bosonix That would be convenient. Do you have a source for this information? – Cryostat

this is specified in the MSDN page msdn.microsoft.com/en-us/library/… and I also looked at the disassembled code. – Pessimism

@bosonix I see. It is stated rather unambigious there, and if the code matches, this seems to be the best solution. I am confused why I did not notice that method (apparently it was available at the time I asked this question), and even Marc Gravell did not suggest it. – Cryostat

Answer 1

A slightly more readable version:

int offset = 0;
while (offset < count)
{
    int read = stream.Read(buffer, offset, count - offset);
    if (read == 0)
        throw new System.IO.EndOfStreamException();
    offset += read;
}

Or written as an extension method for the Stream class:

public static class StreamUtils
{
    public static byte[] ReadExactly(this System.IO.Stream stream, int count)
    {
        byte[] buffer = new byte[count];
        int offset = 0;
        while (offset < count)
        {
            int read = stream.Read(buffer, offset, count - offset);
            if (read == 0)
                throw new System.IO.EndOfStreamException();
            offset += read;
        }
        System.Diagnostics.Debug.Assert(offset == count);
        return buffer;
    }
}

Pessimism

Answer 2

Simply; you loop;

int read, offset = 0;
while(leftToRead > 0 && (read = stream.Read(buf, offset, leftToRead)) > 0) {
    leftToRead -= read;
    offset += read;
}
if(leftToRead > 0) throw new EndOfStreamException(); // not enough!

After this, buf should have been populated with exactly the right amount of data from the stream, or will have thrown an EOF.

Jari

this is not java, throw by value, do not use new operator – Inspired

@Jiří I would love for you to expand on that, because yes: you're meant to use new when throwing in C# (unless you're re-throwing); the only common change I'd expect to see here is to use a utility method for that part instead of having the throw in the same method, so that the throw doesn't disable inlining (which adds an extra stack-frame into the exception, but ... meh, you're already throwing). Are you perhaps thinking of C/C++? – Jari

Yeah, thought this was CPP didn't notice that flag, sorry about that - was just browsing some cpp questions to find solution and noticed this... – Inspired

Answer 3

Getting everything together from answers here I came up with the following solution. It relies on a source stream length. Works on .NET core 3.1

/// <summary>
/// Copy stream based on source stream length
/// </summary>
/// <param name="source"></param>
/// <param name="destination"></param>
/// <param name="bufferSize">
/// A value that is the largest multiple of 4096 and is still smaller than the LOH threshold (85K).
/// So the buffer is likely to be collected at Gen0, and it offers a significant improvement in Copy performance.
/// </param>
/// <returns></returns>
private async Task CopyStream(Stream source, Stream destination, int bufferSize = 81920)
{
    var buffer = new byte[bufferSize];
    var offset = 0;
    while (offset < source.Length)
    {
        var leftToRead = source.Length - offset;
        var lengthToRead = leftToRead - buffer.Length < 0 ? (int)(leftToRead) : buffer.Length;
        var read = await source.ReadAsync(buffer, 0, lengthToRead).ConfigureAwait(false);
        if (read == 0)
            break;
        await destination.WriteAsync(buffer, 0, lengthToRead).ConfigureAwait(false);
        offset += read;
    }
    destination.Seek(0, SeekOrigin.Begin);
}

Mycenae