GZipStream works when writing to FileStream, but not MemoryStream

Question

If compress some json text, and write that it to a file using a FileStream I get the expected results. However, I do not want to write to disk. I simply want to memorystream of the compressed data.

Compression to FileStream:

string json = Resource1.json;

using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (FileStream output = File.Create(@"C:\Users\roarker\Desktop\output.json.gz"))
{
    using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
    {
        input.CopyTo(compression);
    }
}

Above works. Below, the output memory stream is length 10 and results in an empty .gz file.

string json = Resource1.json;

using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (MemoryStream output = new MemoryStream())
{
    using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
    {
        input.CopyTo(compression);

        byte[] bytes = output.ToArray();
    }
}

EDIT: Moving output.ToArray() outside the inner using clause seems to work. However, this closes the output stream for most usage. IE:

        using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
        using (MemoryStream output = new MemoryStream())
        {
            using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
            {
                input.CopyTo(compression);
            }
            WriteToFile(output);
        }

where :

    public static void WriteToFile(Stream stream)
    {
        using (FileStream output = File.Create(@"C:\Users\roarker\Desktop\output.json.gz"))
        {
            stream.CopyTo(output);
        }
    }

This will fail on stream.CopyTo because the stream has been closed. I know I could make a new Stream from bytes of output.ToArray(), but why is this necessary? why does ToArray() work when the stream is closed?

Final Edit:

Just needed to use the contructor of the GZipStream with the leaveOpen parameter.

Ignorance

To be clear, bytes.Length == 10 in the end. – Ignorance

Do you get the same behavior if you move byte[] bytes = output.ToArray(); outside the inner using block? – Clannish

Streams are like toilets. You're not done with them until they're flushed. Unless you're some kind of disgusting animal. I mean, really. – Freeland

Related: #34567059 – Heterolysis

Answer 1

You're calling ToArray() before you've closed the GZipStream... that means it hasn't had a chance to flush the final bits of its buffer. This is a common issue for compression an encryption streams, where closing the stream needs to write some final pieces of data. (Even calling Flush() explicitly won't help, for example.)

Just move the ToArray call:

using (MemoryStream input = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (MemoryStream output = new MemoryStream())
{
    using (GZipStream compression = new GZipStream(output, CompressionMode.Compress))
    {
        input.CopyTo(compression);
    }
    byte[] bytes = output.ToArray();
    // Use bytes
}

(Note that the stream will be disposed when you call ToArray, but that's okay.)

Attitudinarian

I'm confused because the output stream is considered closed after the inner using. I can use ToArray() and get all the bytes, but I cannot pass the output to other function to be treated like a memorystream. Does this make sense? – Ignorance

@Dave: Yes - basically you're using the extra functionality of MemoryStream (that it has the data in memory) but you can't use it as a regular stream. It's really, really handy to be able to do this - precisely in this sort of situation. – Attitudinarian

I edit'd the question, but I guess what you're saying is that if I want to access that data in memory, I need to copy those bytes into a new stream? – Ignorance

@Dave: No, I'm not saying that at all. You call ToArray(), and you get a byte array with the contents. Access that however you want to... there's no need to create a new stream with that data unless you want to. If you need it in the form of a stream, then yes, you'd create a new MemoryStream to wrap it. In some cases there are ways to make "wrapping" streams not close the underlying ones, but it's not universal. – Attitudinarian

Ahh, I can probably use the leaveOpen constructor arguement to do what I want. – Ignorance

I thought the whole point of working with Streams is so you don't have to create arrays (which chew up memory). Surely it's possible to pass in a stream and compress the contents to another stream. In my case I need a stream to pass to another method which "streams" the data to azure storage - ideally there would be no arrays in that process. – Throb

@tigerprawn: We can't really tell much about your context here. This answer is about arrays because the question is about arrays. Presumably in your context, you don't have arrays. If you're having difficulty with what you're trying to do, I suggest you ask a new question with a lot more detail. – Attitudinarian