Getting "connection refused" when trying to access etcd from within a Docker container

Question

I am trying to access etcd from within a running Docker container. When I run

curl http://172.17.42.1:4001/v2/keys

I get

curl: (7) Failed to connect to 172.17.42.1 port 4001: Connection refused

I have four other hosts where this works fine, but every container on this machine has this problem. I'm really at a loss as to what's going on, and I don't know how to debug it.

My etcd environment variables are

ETCD_ADVERTISE_CLIENT_URLS=http://10.242.10.2:2379
ETCD_DISCOVERY=https://discovery.etcd.io/<token_removed>
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.242.10.2:2380
ETCD_LISTEN_CLIENT_URLS=http://10.242.10.2:2379,http://127.0.0.1:2379,http://0.0.0.0:4001
ETCD_LISTEN_PEER_URLS=http://10.242.10.2:2380

I can also access etcd from the host with

curl http://localhost:4001/v2/keys

So there seems to be some error when routing from the container out to the host. But I can't figure out what it is. Can anyone point me in the right direction?

Clipping

Are you sure etcd is listening on that IP? 172.17.42.1 looks like the docker0 ip. Use ss -l to list open server sockets. I guess etcd is listening on a different IP. – Seaver

Hmm you're right. Etcd is only listening on 127.0.0.1:4001, even though I have 0.0.0.0:4001 in my config. I'm not sure why it's not respecting that. – Clipping

Answer 1

It turns out etcd was only listening on localhost:4001 on that machine, which is why I couldn't access it from within a container. This is despite me configuring one of the listen client urls to 0.0.0.0:4001.

It turns out that I had run sudo systemctl enable etcd2, which caused it to run before the cloud-config service ran. As such, etcd started with default configuration instead of the one that I had specified in my cloud config. Running sudo systemctl disable etcd2 fixed the issue.

Clipping

Answer 2

I observed I had to use the --advertise-client-urls and --listen-client-urls. Like so:

./etcd --advertise-client-urls 'http://0.0.0.0:2379,http://0.0.0.0:4001' --listen-client-urls 'http://0.0.0.0:2379,http://0.0.0.0:4001'

Then I was able to successfully do

curl -L http://hostname:2379/version

from any machine that could reach that server and it worked.

Theis

Can't upvote this enough. Wish this was pointed out as required for running etcd in docker. Thanks. – Lines