Go's sync.WaitGroup lost the one of the responses

Question

I'm trying to send http requests sequently by adding time.Sleep inside goroutine by myself.

However, there's always lost one of the response by sync.WaitGroup, for example, this following go client sent 5 requests to my web server, but only got 4 of the total 5 responses:

Sending http://localhost:9001/?id=1, at 2018-06-11 17:11:56.424086867 +0800 CST m=+0.000949479
Sending http://localhost:9001/?id=2, at 2018-06-11 17:11:57.426178028 +0800 CST m=+1.003040640
GOT id: 2 sleeping .... 0.347917120258,  at: 2018-06-11 17:11:57.776187964 +0800 CST m=+1.353050576
GOT id: 1 sleeping .... 1.63133622383,  at: 2018-06-11 17:11:58.059441646 +0800 CST m=+1.636304258
Sending http://localhost:9001/?id=3, at 2018-06-11 17:11:58.42641506 +0800 CST m=+2.003277672
GOT id: 3 sleeping .... 0.959551004983,  at: 2018-06-11 17:11:59.392013618 +0800 CST m=+2.968876230
Sending http://localhost:9001/?id=4, at 2018-06-11 17:11:59.428900219 +0800 CST m=+3.005762831
GOT id: 4 sleeping .... 0.0479890727854,  at: 2018-06-11 17:11:59.479683953 +0800 CST m=+3.056546565
Sending http://localhost:9001/?id=5, at 2018-06-11 17:12:00.428293512 +0800 CST m=+4.005156124

Here's the Go client code

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "sync"
    "time"
)

func main() {
    urls := []string{
      "http://localhost:9001/?id=1",
      "http://localhost:9001/?id=2",
      "http://localhost:9001/?id=3",
      "http://localhost:9001/?id=4",
      "http://localhost:9001/?id=5",
    }
    jsonResponses := make(chan string)

    var wg sync.WaitGroup

    wg.Add(len(urls))

    for i, url := range urls {
        tsleep := i
        go func(url string) {
            defer wg.Done()
            time.Sleep(time.Duration(tsleep) * time.Second)
            fmt.Println("Sending " + url + ", at " + time.Now().String())
            res, err := http.Get(url)
            if err != nil {
                log.Fatal(err)
            } else {
                defer res.Body.Close()
                body, err := ioutil.ReadAll(res.Body)
                if err != nil {
                    log.Fatal(err)
                } else {
                    t := time.Now()
                    jsonResponses <- string("GOT id: " + string(body) + ",  at: " + t.String())
                }
            }
        }(url)
    }

    go func() {
        for response := range jsonResponses {
            fmt.Println(response)
        }
    }()

    wg.Wait()
}

With my testing tornado python web server code

import tornado.ioloop
import tornado.web
import random
import tornado.gen

class DefaultHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        id = self.get_query_argument("id", "1")
        sleepy = 2.0 * (random.random())
        self.write(id + " sleeping .... " + str(sleepy))
        yield tornado.gen.sleep(sleepy)
        self.finish()


def make_app():
    return tornado.web.Application([
        (r"/", DefaultHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(9001)
    tornado.ioloop.IOLoop.current().start()

Installment

Answer 1

wg.Wait() will only wait until all the goroutines that make the HTTP calls are done, it does not wait for the goroutine that prints the results to complete. And when all HTTP calls are done (and their results are sent on the channel), wg.Wait() may return, and your main() function ends. And with it your app ends as well. It will not wait for the independent, concurrent goroutine printing the results to end.

To make your app wait for that too, use a 2nd WaitGroup or other means for synchronization. And don't forget to close the jsonResponses channel once all HTTP calls are completed, as that will make the printing goroutine end (once all values are received before it was closed):

var wg2 sync.WaitGroup
wg2.Add(1)
go func() {
    defer wg2.Done()
    for response := range jsonResponses {
        fmt.Println(response)
    }
}()

wg.Wait()

// At this point HTTP calls are done.
// Close jsonResponses, signalling no more data will come:
close(jsonResponses)
wg2.Wait()

What happens here is that once wg.Wait() reutrns, we know all HTTP calls and delivering their results have completed. We may close jsonResponses here. And the for range loop in the printing goroutine will properly terminate once all values sent before the channel close are received. And finally it will call wg2.Done(), so the wg2.Wait() call in the main can return and your program end.

Ormuz

Thanks! I'm a bit of a go newb. I had tried your solution -- except I forgot to close the channel, which caused the whole thing to hang. Thanks for pointing that out! – Anarchic