Concourse cannot reach Docker registry
Asked Answered
S

2

6

I've set up a Concourse server (on bare metal), and everything is working swimmingly, except that I can't start any jobs and at this point I'm at a complete loss. Any help or suggestions would be greatly appreciated!

Taking the tutorial's hello world as an example, I get the following error:

$ fly -t tutorial execute -c task_hello_world.yml
executing build 55 at https://{{full hostname}}/builds/55 
initializing
resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to ping registry: 2 error(s) occurred:

* ping https: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
* ping http: Get http://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

errored

Grepping through the logs for any mention of docker, I get the following (and no other errors in the syslog):

Jan 11 03:39:11 {{host}} concourse[13704]: {"timestamp":"1515641951.825016260","source":"worker","message":"worker.garden.extract-resources.extract.already-extracted","log_level":1,"data":{"resource-type":"docker-image","session":"2.1.6"}}

Jan 11 03:41:55 {{host}} concourse[13613]: {"timestamp":"1515642115.660775423","source":"atc","message":"atc.create-build.do.task.image.failed-to-get-latest-image-version","log_level":2,"data":{"container":"ce9a7c49-210d-4cf7-6786-d27b1046c76b","error":"resource script '/opt/resource/check []' failed: exit status 1\n\nstderr:\nfailed to ping registry: 2 error(s) occurred:\n\n* ping https: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n* ping http: Get http://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n","session":"240.3.1.2"}}

I've started my web target with:

/usr/share/concourse/bin/concourse \
  web \
  --basic-auth-username {{username}} \
  --basic-auth-password {{password}} \
  --session-signing-key /usr/share/concourse/keys/session_signing_key \
  --tsa-host-key /usr/share/concourse/keys/tsa_host_key \
  --tsa-authorized-keys /usr/share/concourse/keys/authorized_keys \
  --bind-port 8080 \
  --tls-bind-port 8443 \
  --tls-key /etc/letsencrypt/live/{{full hostname}}/privkey.pem \
  --tls-cert /etc/letsencrypt/live/{{full hostname}}/fullchain.pem \
  --postgres-data-source postgres://concourse:concourse@localhost/atc \
  --external-url https://{{full hostname}}

...and my worker with:

/usr/share/concourse/bin/concourse \
  worker \
  --garden-dns-server 8.8.8.8 \
  --work-dir /usr/share/concourse/workspace \
  --tsa-host {{full hostname}} \
  --tsa-public-key /usr/share/concourse/keys/tsa_host_key.pub \
  --tsa-worker-private-key /usr/share/concourse/keys/worker_key

Docker appears to work fine (e.g., docker run hello-world), as does logging in with docker login. The worker nodes seem happy, and even register that they know about some containers (!?):

$ fly -t tutorial workers
name    containers platform  tags  team  state    version 
worker  3           linux    none  none  running  1.2

I have iptables routing ports 80 and 443 to 8080 and 8443, but there doesn't appear to be any difference if I turn this off and use the secondary ports. Curl seems to work fine, which I take to suggest that it's not a DNS issue:

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Again, any suggestions are appreciated!

Stockjobber answered 11/1, 2018 at 3:59 Comment(5)
seems similar with this issue github.com/concourse/concourse/issues/374Jadejaded
Yeah, I saw that, but it doesn't seem to help. I'm not running in a container or on a corp network, and DNS seems to work fine. For good measure, I edited resolv.conf to point to 8.8.8.8 (the error persists), and checked DNS again: $ nslookup registry-1.docker.io Server: 8.8.8.8 Address: 8.8.8.8#53 Non-authoritative answer: Name: registry-1.docker.io Address: 34.200.28.105 ...Stockjobber
Situation is complicated. Do not add 8.8.8.8 to resolv.conf, add this flag to the concourse worker cli invocation: "--garden-dns-server 8.8.8.8". Does it work ? The full explanation is by reading the tens of tickets opened on concourse github for this problem... The UNAUTHORIZED you get when using curl is misleading, I went through the same rabbit hole :-)Accommodative
Did you manage to get anywhere with this in the end, @GaborAngeli?Ahn
No, I switched to self hosted Gitlab and life became instantly much better.Stockjobber
I
6

I have been facing this issue for a while and tried almost everything like disabling firewall,changing iptables settings.

Finally while playing around with docker-compose.yml file to set up concourse, I changed the following 2 settings in the file and resolved the issue.

CONCOURSE_GARDEN_DNS_PROXY_ENABLE=true
CONCOURSE_WORKER_GARDEN_DNS_PROXY_ENABLE=true

Hope this resolves your issue as well.

Incredible answered 29/4, 2019 at 19:45 Comment(0)
L
2

I had this same issue on an Ubuntu VM, and it turned out that my iptables chain FORWARD policy was set to DROP. Changing the FORWARD policy to ACCEPT resolved the issue. $ iptables -P FORWARD ACCEPT

Laos answered 15/10, 2018 at 14:1 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.