Golang unexpected EOF

Question

Here's my code, I'm new to Go. I tried googling the issue, but I can't quite put my finger on it. I think it has something to do with the Read() method.

package main

import (
    ...
)

type compressor struct {
    content []byte
}

func (r *compressor) compress() []byte {
    ...
}

func (r *compressor) decompress() []byte {
    var buffer bytes.Buffer
    dc := flate.NewReader(&buffer)
    _, err := dc.Read(r.content)
    if err != nil {
        if err != io.EOF {
            log.Fatal(err)
        }
    }

    return buffer.Bytes()
}

func main() {
    fileName := os.Args[1]
    fmt.Println(os.Args)
    contents, err := ioutil.ReadFile(fileName)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Print("Uncompressed data: ")
    fmt.Println(len(contents))

    comp := compressor{contents}
    buffer := comp.decompress()
    fmt.Print("Uncompressed data: ")
    fmt.Println(len(comp.decompress()))

    err = ioutil.WriteFile(fileName+".decjc", buffer, 0644)
    if err != nil {
        log.Fatal(err)
    }
}

Here's the output

dylan@skynet:~/Documents/EXP/jc$ ./jc data.txt.jc 
[./jc data.txt.jc]
Uncompressed data: 2364480
2018/06/29 21:41:35 unexpected EOF

Lambent

Where is the error being thrown? What line? – Kellsie

You're reading from an empty buffer, so there's nothing to decompress. – Zhdanov

The buffer actually does contain content – Lambent

Perhaps your first line in decompress() you meant bytes.NewReader(r.content) – Beseem

@Dylan: no it does not. You're reading from buffer which is a bytes.Buffer which you never put any data into. – Zhdanov

Have another look at io.Reader.Read. It reads into its argument, not from it. Where it's reading from depends entirely on the implementation. flate's reader reads from the thing you pass to NewReader. – Succotash

Please take a look at this Github issue comment: github.com/revel/revel/issues/566#issuecomment-42019967 – Judicial

@andy: Thanks! I ran into the same issue and the comment you linked is the true solution. – Gyroscope

Answer 1

After doing a trace on the particular code in question I have come to the following answer.

/src/bytes/reader.go 70

func (r *Reader) ReadByte() (byte, error) {
    ...

    if r.i >= int64(len(r.s)) {
        return 0, io.EOF
    }

    ....
}

There are four functions in bytes/reader that can return io.EOF, and zero functions that can return io.ErrUnexpectedEOF. The four functions that can return io.EOF are:

Read(b []byte)
ReadAt(b []byte, off int64)
ReadByte()
ReadRune()

/src/compress/flate/inflate.go 698

func (f *decompressor) moreBits() error {
    c, err := f.r.ReadByte()
    if err != nil {
        return noEOF(err)
    }

    ...
}

Of the four functions that can return io.EOF, only one function in flate/inflate.go calls any of them: moreBits() calls ReadByte()

/src/compress/flate/inflate.go 690

func noEOF(e error) error {
    if e == io.EOF {
        return io.ErrUnexpectedEOF
    }

    ...
}

When moreBits() receives an error it calls noEOF(), which checks if it had received an io.EOF. If this was the case then io.ErrUnexpectedEOF is returned backed. Everything seems to be working as intended, and it appears that it is the user's responsibility to be on the look out for this particular case. A suggested edit to the code above to handle what appears to be defined behavior is:

func (r *compressor) decompress() []byte {
    dc := flate.NewReader(bytes.NewReader(r.content))
    defer dc.Close()
    rb, err := ioutil.ReadAll(dc)
    if err != nil {
        if err != io.EOF && err != io.ErrUnexpectedEOF {
            log.Fatalf("Err %v\n read %v", err, rb)
        }
    }
    return rb
}

This was checked under go1.12.9

Geologize

I ran in this problem in a test and your solution worked there. I don't think it's right, though, but for a test, I can have a few hacks like this. – Invagination

Answer 2

You got the in and outputs mixed up.

flate.NewReader takes the compressed input as an io.Reader and it returns a io.ReadCloser that can be used to get the uncompressed output:

func (r *compressor) decompress() []byte {
    dc := flate.NewReader(bytes.NewReader(r.content))
    defer dc.Close()
    rb, err := ioutil.ReadAll(dc)
    if err != nil {
        if err != io.EOF {
            log.Fatalf("Err %v\n read %v", err, rb)
        }
    }
    return rb
}

Cumbersome

Reading into an empty byte slice is futile. You probably meant ioutil.ReadAll or so. – Succotash

another thing, i found out, is that something goes wrong in the encoding while compressing. So while decompressing one of the bytes get decoded to an EOF. – Lambent

note that compress/flate does not uncompress gzip files. there is compress/gzip for that. – Cumbersome