net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Question

I am getting this problem while pushing the images to registry.

I'm using Ubuntu server inside Ubuntu docker installed, and inside the docker created docker registry container and I have an issue while while registry is integrating with gitlab.

Both gitlab and registry are on same server.

I have created a registry under same domain of gitlab.

no code

Get https://someurl:8000/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Osullivan

Timeouts happend when the target is not reachable, have you checked if the host can be reached by your browser or using ping/curl on the server? – Rhetor

its pinging but while when i am pushing images to registry it is showing the error @Rhetor – Osullivan

Could you try pushing with http instead of https the library probably doesn't trust the certificate. @AbhayChowdary – Rhetor

@Rhetor k sure i will try thanks for the reply – Osullivan

Check this solution here github.com/docker/for-win/issues/611#issuecomment-531764867 – Bashan

Answer 1

I was getting a similar error message:

Get http://localhost:5000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

when doing a docker push command on my localhost:

docker push localhost:5000/redis:3.0.7

I saw an interesting point there and tried it.

After I edited the /etc/hosts file and commented out the line:

#::1    localhost ip6-localhost ip6-loopback

the issue was gone.

Rosarosabel

This worked for me, but I also had to comment out two more lines in /etc/hosts, namely: ff02::1 ip6-allnodes and ff02::2 ip6-allrouters. Additionally, since my private registry is behind an nginx load balancer, I had to add the fully-qualified domain name of my nginx server to /etc/hosts, meaning I added the line 127.0.1.1 registry.mydomain.com – Redblooded

NOTE: In Docker, the /etc/hosts file is located inside the container, not on the host system. The /etc/hosts file in a container contains mappings of IP addresses to hostnames, similar to the /etc/hosts file on a regular operating system. Is this correct? – Derrik

Answer 2

This error encountered due to the destination unreachability problem. It means that the port might be filtered by the local or external firewall or as another answer, the DNS setting of the machine might be the trouble.

Therefore, make sure you're connecting to the indeed destination. My problem was the firewall of the service provider that I got the server, after I created a ssh tunnel, I've logged in as local.

Follow the procedure to check what's the problem.

$ sudo docker login -u mrt https://gitlab.mydomain.com:5050

Error response from daemon: Get https://gitlab.mydomain.com:5050: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Check the connection:

$ nmap -p 5050 gitlab.mydomain.com

Starting Nmap 7.80 ( https://nmap.org ) at 2020-10-12 14:15 +0330
Nmap scan report for gitlab.mydomain.com (x.x.x.x)
Host is up (0.057s latency).
rDNS record for x.x.x.x: x.x.x.x

PORT     STATE     SERVICE
5050/tcp filtered  mmcc

Nmap done: 1 IP address (1 host up) scanned in 0.27 seconds

Create an ssh tunnel:

$ ssh -p 26 -L 5050:gitlab.mydomain.com:5050 [email protected]

Then:

$ sudo docker login -u mrt https://localhost:5050

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Neoteny

Answer 3

I was getting the same error (on Windows):

In my case, it was an issue with the proxy. So I:

• started px (NTLM would also make it) configured on http://localhost:3128
• configured the proxy in Docker > Settings > Resources :

This solved the error:

Masjid

Use kernel networking for UDP on the Network tab was helpful – Perithecium

Answer 4

(For private company networks only)

Got the following error intermittently:

net/http request canceled (client.timeout exceeded while awaiting headers)

because DNS was configured in the /etc/docker/daemon.json and the request was failing every time through the public IP i.e. 8.8.8.8. We added another company nameserver and the issue was resolved.

From:

{ "dns" : [ "114.114.114.114" , "8.8.8.8" ] }

To:

{ "dns" : [ "114.114.114.114" , "110.110.110.110", "8.8.8.8" ] }

Amorist

Answer 5

On Ubuntu 20 I had to open the 5050 port like so: sudo ufw allow 5050/tcp

To see if this is your issue you can run install nmap on gitlab and check to see if port 5050 is open:

apt install nmap
nmap localhost

My results were:

root@gitlab:~# nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2022-07-25 09:04 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000070s latency).
Not shown: 988 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
443/tcp  open  https
3000/tcp open  ppp
5000/tcp open  upnp
5050/tcp open  mmcc
8080/tcp open  http-proxy
8082/tcp open  blackice-alerts
8090/tcp open  opsmessaging
9090/tcp open  zeus-admin
9100/tcp open  jetdirect

So I had port 5050 open.

Then I ran nmap from my local machine

❯ nmap gitlab.example.com
Starting Nmap 7.92 ( https://nmap.org ) at 2022-07-25 11:52 EEST
Nmap scan report for gitlab.example.com (164.92.252.91)
Host is up (0.028s latency).
Not shown: 997 filtered tcp ports (no-response)
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

as you can see we are missing port 5050.

Running sudo ufw allow 5050/tcp on Ubuntu 20 solved the issue for me

Smithery

Answer 6

I had similar failure, but for default docker repository:

ERROR: Get "https://registry-1.docker.io/v2/": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

and

> [internal] load metadata for docker.io/library/openjdk:8-jre:
------
failed to solve: rpc error: code = Unknown desc = failed to solve
with frontend dockerfile.v0: failed to create LLB definition: failed
to do request: Head "https://registry-1.docker.io/v2/library/openjdk/manifests/8-jre":
unable to connect to 52.55.168.20:443. Do you need an HTTP proxy?

Finally I've found this record in my /etc/hosts:

52.55.168.20  registry-1.docker.io

No idea which way it was added (may be it was Docker Desktop setup; I do not remember to do it intendedly), but just deleting or commenting this line solved my problem.

Dinar

Answer 7

In my case it was IPv6 that caused DNS instability on my GitLab server. Disabling IPv6 on the server helped:

cat <<EOF >> /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
EOF

Apply settings. sudo sysctl --system

Puissant

Answer 8

Deleting all other DNS server settings except 8.8.8.8 resolved the issue for me.

On mac OS:

Lachman

Answer 9

On Windows OS, Running Linux Containers with WSL2, i followed the following steps,

1. Ran the command docker logout
2. Ran the command, docker network prune, so as to remove all the preconfigured settings of the network.
3. From Docker Settings, Enabled the DNS server configuration with 8.8.8.8
4. Restarted the Docker
5. Now executed login command with registry to login, docker login {registry}

Spectacular

Answer 10

For those who use the docker-compose, particularly on ubuntu:

docker-compose push

That means does not include SUDO. I used: sudo docker-compose push and got the error

Extraction

Answer 11

if in the case of network changes:

ERROR: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Solution:

docker network prune

Ambulacrum

Answer 12

I was using wsl in Windows 10 and faced this issue. I know it's not reasonable but everything was ok before changing the Windows password (but I didn't restart the PC). After restarting the PC and running wsl I could pull the image without this error.

Casas

Answer 13

Solution which I found for this issue is uninstall docker older version and reinstall

Effluent

Answer 14

In my case it was due to multiple entries of registry server in the host file on Windows machine.

I was having:

192.168.1.23 myregistry.local
10.1.25.40 myregistry.local

out of this one was correct.

But still image download used to continue for sometimes and fail after 30 to 50 % with error:

Error response from daemon: Get "https://myregistry.local:5000/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

resolved by deleting the duplicate entry.

Overall the error is very generic and mostly caused due to the HTTP connection failure in docker client connecting to registry.

Griffey