Dial tcp I/O timeout on simultaneous requests

I am building a tool in Go that needs to make a very large number of simultaneous HTTP requests to many different servers. My initial prototype in Python had no problem doing a few hundred simultaneous requests.

However, I have found that in Go this almost always results in a Get http://www.google.com: dial tcp 216.58.205.228:80: i/o timeout for some if the number of simultaneous requests exceeds ~30-40.

I've tested on macOS, openSUSE, different hardware, in different networks and with different domain lists, and changing the DNS server as described in other Stackoverflow answers does not work either.

The interesting thing is that the failed requests do not even produce a packet, as can be seen when checking with Wireshark.

Is there anything that I am doing wrong or is that a bug in Go?

Minimum reproducible program below:

package main

import (
    "fmt"
    "net/http"
    "sync"
)

func main() {
    domains := []string{/* large domain list here, eg from https://moz.com/top500 */}

    limiter := make(chan string, 50) // Limits simultaneous requests

    wg := sync.WaitGroup{} // Needed to not prematurely exit before all requests have been finished

    for i, domain := range domains {
        wg.Add(1)
        limiter <- domain

        go func(i int, domain string) {
            defer func() { <-limiter }()
            defer wg.Done()

            resp, err := http.Get("http://"+domain)
            if err != nil {
                fmt.Printf("%d %s failed: %s\n", i, domain, err)
                return
            }

            fmt.Printf("%d %s: %s\n", i, domain, resp.Status)
        }(i, domain)
    }

    wg.Wait()
}

Two particular error messages are happening, a net.DNSError that does not make any sense and a non-descript poll.TimeoutError:

&url.Error{Op:"Get", URL:"http://harvard.edu", Err:(*net.OpError)(0xc00022a460)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc000aca200)}
&net.DNSError{Err:"no such host", Name:"harvard.edu", Server:"", IsTimeout:false, IsTemporary:false}

&url.Error{Op:"Get", URL:"http://latimes.com", Err:(*net.OpError)(0xc000d92730)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*poll.TimeoutError)(0x14779a0)}
&poll.TimeoutError{}

Update:

Running the requests with a seperate http.Client as well as http.Transport and net.Dialer does not make any difference as can be seen when running code from this playground.

I think many of your net.DNSErrors are actually too many open files errors in disguise. You can see this by running your sample code with the netgo tag (recommendation from here) (go run -tags netgo main.go) which will emit errors like:

…dial tcp: lookup buzzfeed.com on 192.168.1.1:53: dial udp 192.168.1.1:53: socket: too many open files

instead of

…dial tcp: lookup buzzfeed.com: no such host

Make sure you're closing the request's response body (resp.Body.Close()). You can find more about this specific problem at What's the best way to handle "too many open files"? and How to set ulimit -n from a golang program?. (On my machine (macOS), increasing file limits manually seemed to help, but I don't think it's a good solution since it doesn't really scale, and I'm not sure how many open files you'd need overall.)

As suggested by @liam-kelly, I think the i/o timeout error is coming from a DNS server or some other security mechanism. Setting a custom (bad) DNS server IP gives me the same error.

Recommended topics

Hot tags