How do I read a streaming response body using Golang's net/http package?

Question

I am trying to connect to an endpoint that does http streaming of json data. I was wondering how to perform a basic request using Go's net/http package and read the response as it comes in. Currently, I am only able to read the response when the connection closes.

resp, err := http.Get("localhost:8080/stream")
if err != nil {
    ...
}
...
// perform work while connected and getting data

Any insight would be greatly appreciated!

Thanks!

-RC

Raffinose

Did you see this answer and does it help? – Feminize

That code will operate on a stream. If you're not getting it when you expect, it's because of how the server is sending it. I've parsed many GBs of responses with this model. – Mikkimiko

Answer 1

The way to do streaming JSON parsing is with a Decoder:

json.NewDecoder(resp.Body).Decode(&yourStuff)

For a streaming API where it's a bunch of objects coming back (a la Twitter), that should stream great with this model and the built-in encoding/json API. But if it's a large response where you have an object that's got a giant array with 10MB of stuff, you probably need to write your own Decoder to pull those inner pieces out and return them. I'm running into that problem with a library I've written.

Satellite

Thanks for the answer! I managed to get it working by creating a bufio.Reader and calling ReadBytes() on the resp.Body. – Raffinose

Eve, where documentation about max 10MB? i need information – Djerba

It's not a max, it's just the way the parser works, objects at a time. – Satellite

Is there a way to listen for when the response has finished being written? – Ifc

Answer 2

The answer provided by Eve Freeman is the correct way to read json data. For reading any type of data, you can use the method below:

resp, err := http.Get("http://localhost:3000/stream")
...

reader := bufio.NewReader(resp.Body)
for {
    line, err := reader.ReadBytes('\n')
    ...

    log.Println(string(line))
}

Raffinose

But how does one detect EOF on that []byte? – Jubbulpore

EOF is returned with err – Lisalisabet

now using these chunks/lines how can I directly write it to file. Means how write using stream not whole as file. ? – Albanian