How to read multiple times from same io.Reader

Question

I want to use request.Body(type io.ReadCloser) which is containing a image.

I dont want to use ioutil.ReadAll() as i want to write this body directly to the file as well as want to decode it, so i only want to use the reference to the content to pass to further function calls,

I tried creating multiple instances of reader for example shown below

package main

import (
    "io/ioutil"
    "log"
    "strings"
)


func main() {
    r := strings.NewReader("some io.Reader stream to be read\n")
    a := &r
    b := &r
    log.Println(ioutil.ReadAll(*a))
    log.Println(ioutil.ReadAll(*b))

}

but in second call it always results into nil.

Please help me how can i pass multiple separate reference for the same reader?

Lophophore

Undoable the way you asked for. Use a io.TeeReader to capture what you read for further reference, e.g. in a bytes.Buffer. – Merlynmermaid

Answer 1

io.Reader is treated like a stream. Because of this you cannot read it twice. Imagine an incoming TCP connection - you cannot rewind what's coming in.

But you can use the io.TeeReader to duplicate the stream:

package main

import (
    "bytes"
    "io"
    "io/ioutil"
    "log"
    "strings"
)

func main() {
    r := strings.NewReader("some io.Reader stream to be read\n")
    var buf bytes.Buffer
    tee := io.TeeReader(r, &buf)

    log.Println(ioutil.ReadAll(tee))
    log.Println(ioutil.ReadAll(&buf)) 
}

Example on Go Playground

Edit: As @mrclx pointed out: You need to read from the TeeReader first, otherwise the buffer will be empty.

Scraper

Also, if the stream's data will be large - say 1GB - don't use ReadAll. Instead - and since the OP wants to write it first to a file - use err, _ := io.Copy(fh, tee) – Mceachern

Answer 2

When you read from ioutil.ReadAll(r) then, the content is gone. You can’t read from it a second time. For an example:

var response *http.Response

//Read the content
rawBody, err := ioutil.ReadAll(response.Body)
    if err != nil {
        t.Error(err)
    }

// Restore the io.ReadCloser to it's original state
response.Body = ioutil.NopCloser(bytes.NewBuffer(rawBody))

Plumbum

Answer 3

When you call ReadAll it's going to empty the buffer, so the second call will always return nothing. What you could do is save the result of ReadAll and reuse that in your functions. For example:

bytes, _ := ioutil.ReadAll(r);
log.Println(string(bytes))

Antisemite

A bad method if the reader contains VERY LARGE data, such as 2G byes. – Topographer

As the name says, it's "ReadAll" so indeed you need to make sure you don't run out of memory. In many cases ReadAll is fine - no need for the complexity of streams if you only have a few MBs to read for example – Antisemite

Answer 4

Technically, on one reader, you cannot read multiple times.

• Even if you create different references but
• when you read once it will be same object referred by all references.
• so what you can do is read the content and store it in one variable.
• Then use that variable as many times as you want.

This will print twice.

package main

import (
    "io/ioutil"
    "log"
    "strings"
)

func main() {
    r := strings.NewReader("some io.Reader stream to be read\n")
    stringData, _ := ioutil.ReadAll(r)
    log.Println(stringData)
    log.Println(stringData)
}

Vindicate

I want to know the reason why not :-) – Lophophore

@AbhishekSoni: An io.Reader is a stream. Once you finish the stream, there is no more. If you read part of the stream, the next read continues where you left off. – Ryurik

Answer 5

Clone the Reader struct.

package main

import (
    "io/ioutil"
    "log"
    "strings"
)

func main() {
    r := strings.NewReader("some io.Reader stream to be read\n")
    v := new(strings.Reader)
    *v = *r
    log.Println(ioutil.ReadAll(r))
    log.Println(ioutil.ReadAll(v))

}

Automata