docker push to local registry fails

Question

Registry container is running at port 5000.

# docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED         STATUS         PORTS                                       NAMES
accbeafa34fe   ibmcom/registry:2.6.2.5   "registry serve /etc…"   3 minutes ago   Up 3 minutes   0.0.0.0:5000->5000/tcp, :::5000->5000/tcp   local-registry

sudo -E docker push localhost:5000/pgp-bench:latest
The push refers to repository [localhost:5000/pgp-bench]
ca9a5caaa0d5: Pushed 
dd9235795971: Pushed 
4dfdebe31f24: Pushed 
98a59ff53764: Retrying in 1 second

-->on retrying
# docker push localhost:5000/pgp-bench:latest
The push refers to repository [localhost:5000/pgp-bench]
Get http://localhost:5000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

On curl, it keeps waiting

# curl http://localhost:5000

Logs of the registry container

time="2021-10-08T06:25:19.963905582Z" level=info msg="endpoint local-5003 disabled, skipping" environment=development go.version=go1.11.9 ins
tance.id=88f73ea0-77d6-425f-8259-0b3717c76b7a service=registry version=e38b0207                                                              
time="2021-10-08T06:25:19.963921607Z" level=info msg="endpoint local-8083 disabled, skipping" environment=development go.version=go1.11.9 ins
tance.id=88f73ea0-77d6-425f-8259-0b3717c76b7a service=registry version=e38b0207                                                              
time="2021-10-08T06:25:19.964454493Z" level=info msg="using redis blob descriptor cache" environment=development go.version=go1.11.9 instance
.id=88f73ea0-77d6-425f-8259-0b3717c76b7a service=registry version=e38b0207                                                                   
time="2021-10-08T06:25:19.964744256Z" level=info msg="listening on [::]:5000" environment=development go.version=go1.11.9 instance.id=88f73ea
0-77d6-425f-8259-0b3717c76b7a service=registry version=e38b0207  
time="2021-10-08T06:25:59.965390038Z" level=debug msg="filesystem.Stat(\"/\")" 
environment=development go.version=go1.11.9 instance.id=88f73e
a0-77d6-425f-8259-0b3717c76b7a service=registry trace.duration=35.698µs trace.file="/go/src/github.com/docker/distribution/registry/storage/d
river/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=8f6467ca-1438-45cc-9910-6c
ba6845ca83 trace.line=137 version=e38b0207

I have tried all the possible options mentioned in similar issues such as

1. net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2. https://github.com/ubuntu/microk8s/issues/196

but nothing seems to help resolve the issue. Any suggestions on how to approach a solution?

Buddy

What happens if you curl -4 http://localhost:5000/v2/? – Easton

Can you share logs of your container accbeafa34fe while doing such requests? – Bick

@BMitch, on curl, there's no o/p and it just waits. – Buddy

@TruongHua, I have added the logs to the question above since they couldn't be accommodated in the comment character limit. – Buddy

I'd check if a firewall setting is blocking the request. – Easton

On retrial, I see the below message now, # curl -4 http://localhost:5000/v2/ curl: (56) Recv failure: Connection reset by peer # curl -4 http://localhost:5000/v2/ {} First it says the connection is reset by peer and then I see an o/p of empty body. Then it again keeps waiting as before. – Buddy

I checked from within the container if it is listening on port 5000 and it is. – Buddy

And regarding firewall setting -- # ufw status Status: inactive – Buddy

Answer 1

I had this same issue.

I changed the tag name to be 127.0.0.1:5000 instead of localhost and it completed as expected when pushing.

Hoekstra

Answer 2

In my case, local registry was connected to the kind cluster.
docker push worked after I disconnected registry from the network:
docker network disconnect "kind" "kind-registry"

Flowing