Docker cannot resolve DNS on private network [closed]

Question

Docker is a software tool primarily used by programmers as it is the mechanism programmers use to produce container images.

My machine is on a private network with private DNS servers, and a private zone for DNS resolution. I can resolve hosts on this zone from my host machine, but I cannot resolve them from containers running on my host machine.

Host:

root@host:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1

root@host:~# ping privatedomain.io
PING privatedomain.io (192.168.0.101) 56(84) bytes of data.

Container:

root@container:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 8.8.8.8
nameserver 8.8.4.4

root@container:~# ping privatedomain.io
ping: unknown host privatedomain.io

It's fairly obvious that Google's public DNS servers won't resolve my private DNS requests. I know I can force it with docker --dns 192.168.0.1, or set DOCKER_OPTS="--dns 192.168.0.1" in /etc/default/docker, but my laptop frequently switches networks. It seems like there should be a systematic way of solving this problem.

Titled

In my case, since i used dnscrypt-proxy, all I need to do is to change /etc/docker/daemon.json add "dns": ["x.y.z.a"] --> ip from netstat -antu | grep :53 --> find for IP addresses that are not localhost (10.x or 192.x or 172.x) – Devlen

Answer 1

Docker populates /etc/resolv.conf by copying the host's /etc/resolv.conf, and filtering out any local nameservers such as 127.0.1.1. If there are no nameservers left after that, Docker will add Google's public DNS servers (8.8.8.8 and 8.8.4.4).

According to the Docker documentation:

Note: If you need access to a host’s localhost resolver, you must modify your DNS service on the host to listen on a non-localhost address that is reachable from within the container.

The DNS service on the host is dnsmasq, so if you make dnsmasq listen on your docker IP and add that to resolv.conf, docker will configure the containers to use that as the nameserver.

1 . Create/edit /etc/dnsmasq.conf^† and add these lines:

interface=lo
interface=docker0

2 . Find your docker IP (in this case, 172.17.0.1):

root@host:~# ifconfig | grep -A2 docker0
docker0   Link encap:Ethernet  HWaddr 02:42:bb:b4:4a:50  
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0

3 . Create/edit /etc/resolvconf/resolv.conf.d/tail and add this line:

nameserver 172.17.0.1

4 . Restart networking, update resolv.conf, restart docker:

sudo service network-manager restart
sudo resolvconf -u
sudo service docker restart

Your containers will now be able to resolve DNS from whatever DNS servers the host machine is using.

^{† The path may be /etc/dnsmasq.conf, /etc/dnsmasq.conf.d/docker.conf, /etc/NetworkManager/dnsmasq.conf, or /etc/NetworkManager/dnsmasq.d/docker.conf depending on your system and personal preferences.}

Titled

Still being unresolved if you have all of the above, reboot your system and see the magic. – Nation

For whatever reason, my containers can't seem to reach DNSMasq when it's listening on the docker0 interface. I have instead opted to use except-interface=eth0 etc. for all of my normal interfaces, and then have dnsmasq listen on 0.0.0.0. Then it works for all custom networks. – Hatter

Answer 2

For Ubuntu 18.04 - 20.10, and other systems that use systemd < v247 systemd-resolved, it may be necessary to install dnsmasq and resolvconf. systemd-resolved is hard-coded to listen on 127.0.0.53, and Docker filters out any loopback address when reading resolv.conf.

1 . Install dnsmasq and resolvconf.

sudo apt update
sudo apt install dnsmasq resolvconf

2 . Find your docker IP (in this case, 172.17.0.1):

root@host:~# ifconfig | grep -A2 docker0
docker0   Link encap:Ethernet  HWaddr 02:42:bb:b4:4a:50  
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0

3 . Edit /etc/dnsmasq.conf and add these lines:

interface=docker0
bind-interfaces
listen-address=172.17.0.1

4 . Create/edit /etc/resolvconf/resolv.conf.d/tail and add this line:

nameserver 172.17.0.1

5 . Restart networking, update resolv.conf, restart docker:

sudo service network-manager restart
sudo resolvconf -u
sudo service dnsmasq restart
sudo service docker restart

Your containers will now be able to resolve DNS from whatever DNS servers the host machine is using.

Titled

This works great right after I make the changes, but on restart I get dnsmasq: failed to create listening socket for 172.17.0.1: Cannot assign requested address, and have to run service dnsmasq restart manually again for it to work. Did you run into this issue? – Zaria

Change bind-interfaces with bind-dynamic. Looks like docker0 interface is not available when dnsmasq service starts so it fails. This option makes dynamically created interfaces work in the same way as the default. – Bowker

@Bowker This didn't work for me. I get cannot set --bind-interfaces and --bind-dynamic, while I do NOT have the bind-interfaces in .conf file. I've tried also removing any interface= from config without success – Demy

Do you know if this has been fixed in later version of docker with systemd-resolved? I have 16.04 machines that have this problem as they are using dnsmasq. On 18.04 machines with systemd-resolved I don't have to change anything, docker seems to pick up my local DNS servers on 18.04 with systemd-resolved without an issue. – Kallman

The magic line that helped me was sudo apt install dnsmasq resolvconf – Flagon

for anyone who is reading this in 2023: > systemd-resolved is hard-coded to listen on 127.0.0.53 This is no longer the case since systemd v247. Please see DNSStubListenerExtra in resolved.conf. – Reformism

@Reformism I Updated the answer to limit the range of affected versions. Thanks! – Titled

@Titled I think the <v247 clarification should apply to the "hard-coded" part not to the whole answer? – Aliber

Answer 3

As you know, Docker copy host /etc/resolv.conf file to containers but removing any local nameserver.

My solution to this problem is to keep using systemd-resolvd and NetworkManager but add dnsmasq and use it to "forward" Docker containers DNS queries to systemd-resolvd.

Step by step guide:

• Make /etc/resolv.conf a "real" file sudo rm /etc/resolv.conf sudo touch /etc/resolv.conf
• Create file /etc/NetworkManager/conf.d/systemd-resolved-for-docker.conf to tell NetworkManager to inform systemd-resolvd but to not touch /etc/resolv.conf [main] # NetworkManager will push the DNS configuration to systemd-resolved dns=systemd-resolved # NetworkManager won’t ever write anything to /etc/resolv.conf rc-manager=unmanaged
• Install dnsmasq sudo apt-get -y install dnsmasq
• Configure dnsmasq in /etc/dnsmasq.conf for listening DNS queries comming from Docker and using systemd-resolvd name server # Use interface docker0 interface=docker0 # Explicitly specify the address to listen on listen-address=172.17.0.1 # Looks like docker0 interface is not available when dnsmasq service starts so it fails. This option makes dynamically created interfaces work in the same way as the default. bind-dynamic # Set systemd-resolved DNS server server=127.0.0.53
• Edit /etc/resolv.conf to use systemd-resolvd nameserver (127.0.0.53) and the host IP (172.17.0.1) in Docker network # systemd-resolvd name server nameserver 127.0.0.53 # docker host ip nameserver 172.17.0.1
• Restart services sudo service network-manager restart sudo service dnsmasq restart sudo service docker restart

For more info see my post (in spanish) https://rubensa.wordpress.com/2020/02/07/docker-no-usa-los-mismos-dns-que-el-host/

Bowker

Thanks, was quite useful. Posted a IMO simpler variant: https://mcmap.net/q/338226/-docker-cannot-resolve-dns-on-private-network-closed – Heterocyclic

Answer 4

Based on answer from @rubensa, but simpler and more integrated IMHO:

• Install dnsmasq sudo apt-get -y install dnsmasq

• Configure dnsmasq in /etc/dnsmasq.d/docker-dns-fix.conf for listening to DNS queries coming from Docker and using systemd-resolvd name server

  # Use interface docker0
  interface=docker0
  # Explicitly specify the address to listen on
  listen-address=172.17.0.1
  # Looks like docker0 interface is not available when dnsmasq
  # service starts so it fails. This option makes dynamically
  # created interfaces work in the same way as the default.
  bind-dynamic
  # Set systemd-resolved DNS server
  server=127.0.0.53 
  

• Tell Docker to use dnsmasq by editing/creating /etc/docker/daemon.json

  {
    "dns": ["172.17.0.1"]
  }
  

• Restart services

  sudo service dnsmasq restart
  sudo service docker restart
  

Heterocyclic

For the last file he meant /etc/docker/daemon.json – Intrusion

Answer 5

I had problems with the DNS resolver in our docker containers. I tried a lot of different things, and in the end, I just figured that my CentOS VPS in Hostgator didn't have installed by default NetworkManager-tui (nmtui), I just installed and reboot it.

sudo yum install NetworkManager-tui

And reconfigured my resolv.conf with default DNS as 8.8.8.8.

nano /etc/resolv.conf

Legra

Answer 6

If you are using a VPN, the VPN protocol might be appending to outbound packets beyond the configured MTU on your private network.

A typical MTU is 1500.

Try adding this content to /etc/docker/daemon.json

{
  "mtu": 1300,
  "dns": ["<whatever DNS server you need in your private network>"]
}

Then systemctl restart docker.

Katharynkathe

Answer 7

My case with many images from docker hub (nodered, syncthing and another):

• container is running under not-root user
• /etc/resolv.conf inside container has permissions 600 and owned by root

So my solution is very simple

root@container:~# chmod 644 /etc/resolv.conf

Profit! :))

Archaeozoic

Answer 8

I have the same error message in my systemctl status docker. I run a Nextcloud and a nextcloud nginx proxy container and used docker compose to install it. It worked for multiple months without big hickups, but on Friday it wasn't accessable. The server had shut down. I restarted it, my icecast2 instance is working fine and was used this sunday for the service in our church. But the docker containers are gone. docker ps -a doesn't show any, I can't access the nextcloud via docker exec like I would do normally. And I have the error message:

No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb 06 19:04:58 ncxxxxxxxxx dockerd[21551]: time="2022-02-06T19:04:58.894366765Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]

My resolv.conf looks like this:

GNU nano 4.8
/etc/resolv.conf
nameserver 127.0.0.53 
options edns0 trust-ad
search fritz.box

Gondolier

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Liselisetta

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review – Flabbergast

Answer 9

It was enough for Ubuntu 18.04 LTS:

sudo service network-manager restart
sudo resolvconf -u
sudo service dnsmasq restart
sudo service docker restart

Rubeola

Another way for fixing that: it's following: add in /etc/docker/daemon.json: next line: ​` { "dns": ["10.0.0.2", "8.8.8.8"] } Then restart the docker service: ​ sudo service docker restart ` – Rubeola

How is this any more than "turn it off, and turn it on again"? – Titled

hahaha of course! and if we buy a new laptop too... 🤓👍🏽 – Rubeola