Docker push - net/http: TLS handshake timeout

Question

I've deployed a private docker image registry on an AWS EC2 Ubuntu 14.04 instance. The registry is secured using Let's Encrypt certificate.

Unfortunately, I'm getting net/http: TLS handshake timeout for docker push operations that take longer than 300s:

This is the output of the time'd command:

[luqo33@home-pc containers]$ time docker push <my-registry-domain:5000>/nginx                                                    
The push refers to a repository [<my-registry-domain:5000>/nginx]
dda5a806f0b0: Layer already exists
ec35cfccb7f7: Layer already exists
94c1a232bb3f: Layer already exists
6d6b9812c8ae: Layer already exists
695da0025de6: Retrying in 1 second
fe4c16cbf7a4: Pushing [================================================>  ]   119 MB/123 MB
net/http: TLS handshake timeout

real    5m0.847s
user    0m0.097s
sys     0m0.017s

Logs of the regsitry:2 container do not show any errors - other than the notification that there was an unexpected EOF while receiving data. I can also push images that take less than 5min to push without problems.

I'm suspecting that it's a system setting at blame as the timeout happens always once the operations goes beyond 300 seconds. There isn't any load balancer or other proxy. <my-registry-domain:5000> points directly at the server IP.

How can I further investigate and possible remedy this situation?

EDIT

The same happens when I push images to other server providers (DigitalOcean), AWS ECS registry or even Docker Hub! I find it hard to believe that the Docker client would have a built-in handshake timeout of 300s.

I'm thinking that perhaps I should start look for the solution at the network level - with my hardware (wi-fi router) or my ISP.

Anybody has a clue what is happening here?

Davidoff

Answer 1

I got the same issue, this issue is may be from your internet connection, I solved it by decrementing the concurrency uploads (downloads for get) to 1 in dockerd. By using these args:

--max-concurrent-downloads  (default: 3)    Set the max concurrent downloads for each pull

--max-concurrent-uploads    (default: 5)    Set the max concurrent uploads for each push

uploading 5 images in same time may result a timeout if you have a low bandwidth.

https://docs.docker.com/engine/reference/commandline/dockerd/

Rounded

this may be a problem for those that are using Azure Devops with a private registry. Since the daemon setting cannot be changed. – Orientalize

This solution did help. (For Linux-Systemd, set in /usr/lib/systemd/system/docker.service, then restart the service.) The connection to our private registry is painfully slow, and decreasing the number of concurrent uploads helps in utilizing the little available bandwidth. – Blossom

github.com/actions/virtual-environments/issues/2152 – Orientalize

In my case , It was my VPN connection that stopped me – Katabolism

Answer 2

Just adding a separate answer for those who may be dealing with it on a managed build environment like Azure Devops based on https://github.com/actions/virtual-environments/issues/2152#issuecomment-736325518

Change the setting as follows:

sudo sed -i 's/ }/, \"max-concurrent-uploads\": 1 }/' /etc/docker/daemon.json
sudo systemctl restart docker

Orientalize

Answer 3

Disable VPN for any AWS services, including ECS. I had it on auto-enable, and received inside ecs-cli (docker compose to ecs):

ERRO[0042] Error describing service                      error="RequestError: send request failed\ncaused by: Post https://ecs.us-east-1.amazonaws.com/: net/http: TLS handshake timeout" service=1routing
FATA[0042] RequestError: send request failed
caused by: Post https://ecs.us-east-1.amazonaws.com/: net/http: TLS handshake timeout

check your internet connection/routing.

Enterectomy

Answer 4

This is a primitive solution, but one thing that worked for me was commenting out some of the later layers and then uncommenting / pushing new images with one layer added at a time until the whole image succeeded.

Nicotiana