Docker Network Nginx Resolver

Question

I am trying to get rid of deprecated Docker links in my configuration. What's left is getting rid of those Bad Gateway nginx reverse proxy errors when I recreated a container.

Note: I am using Docker networks in bridge mode. (docker network create nettest)

I am using the following configuration snippet inside nginx:

location / {
      resolver 127.0.0.1 valid=30s;
      set $backend "http://confluence:8090";
      proxy_pass $backend;

1. I started a container with hostname confluence on my Docker network with name nettest.
2. Then I started the nginx container on network nettest.
3. I can ping confluence from inside the nginx container
4. confluence is listed inside the nginx container's /etc/hosts file
5. nginx log says send() failed (111: Connection refused) while resolving, resolver: 127.0.0.1:53
6. I tried the docker network default dns resolver 127.0.0.11 from /etc/resol.conf
7. nginx log says confluence could not be resolved (3: Host not found)

Anybody knows how to configure nginx resolver with Docker Networks or an alternative on how to force Nginx to correctly resolve the Docker network hostname?

Costumer

Does not look like this will be solved in any way without a network driver. The cause is Nginx by their need for an ip for a resolver and this can't be linked automatically to any Docker managed DNS resolver. – Costumer

Answer 1

First off, you should be using the Docker embedded DNS server at 127.0.0.11.

Your problem could be caused by 1 of the following:

1. nginx is trying to use IPv6 (AAAA record) for the DNS queries.

   See https://mcmap.net/q/205760/-nginx-doesn-39-t-get-host-from-embedded-dns-in-docker-1-10-1 for the solution.

   Basically something like:

   http {
       resolver 127.0.0.11 ipv6=off;
   }
   

   This is probably no longer a problem with Docker 1.11:

   Fix to not forward docker domain IPv6 queries to external servers (#21396)

2. Take care that you don't accidentally override the resolver configuration directive. In my case I had in the server block resolver 8.8.8.8 8.8.4.4; from Mozilla's SSL Configuration Generator, which was overriding the resolver 127.0.0.11; in the http block. That had me scratching my head for a long time...

Kinnikinnick

Hardcoding the IP address of the Docker DNS resolver (127.0.0.11) shouldn't be an issue, as it won't change. Note that the Docker DNS resolver is only available with user defined networks (docker network create ...). Reference: github.com/docker/docker/issues/22652 – Institute

More official reference for the previous comment: docs.docker.com/engine/userguide/networking/#/… – Institute

I had to add ipv6=off when running nginx in kubernetes – Ebby

How do I check inside the container that Nginx Resolves it ? – Wolfenbarger

@Wolfenbarger Why would it be necessary to perform such a check? – Kinnikinnick

I'm getting errors like 6#6: send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53 I tried both resolver 127.0.0.11; as well as resolver 127.0.0.11 ipv6=off; to no avail. Anyone else figured out? – Santee

@Santee Don't use a hardcoded DNS resolver address. And please make sure you're using a user-defined network. The default network works differently. – Kinnikinnick

Actually, @MarcoRoy was right. The documentation guarantees that the address of the embedded DNS resolver in user-defined networks is always 127.0.0.11 - the documentation has moved to docs.docker.com/v17.09/engine/userguide/networking/… – Kinnikinnick

You should also add to your answer you need to setup a link between the nginx container and the backend service. On my docker compose, I didn't add depends_on: - rs-gateway and it only started to work when I add it. You can also say the 127.0.0.11 IP comes from /etc/resolv.conf and may varies depending of the docker runner. – Newhall

Hey that's me again after 1 year !!! Your configuration is not good for docker swarm : take a look at https://mcmap.net/q/205761/-nginx-does-not-automatically-pick-up-dns-changes-in-swarm (needs to set valid=10s and use set to define a dynamic variable and bypass DNS cache) – Newhall

when I add 'resolver 127.0.0.11 ipv6=off;' Nginx throws this exception: 'host not found in resolver "ipv6=off"' – Tindle

Just using resolver 127.0.0.11 inside server block did the tricky for me. docker + docker-compose separate network from default. – Locarno

Answer 2

Maybe you should check your container's /etc/resolv.conf

It shows your container's correct DNS config and then use that DNS server IP for resolver.

127.0.0.11 does not works in Rancher

Periodicity

Oh sorry, just for additional recommendation. Because my friend complained it did not work, maybe we'd read answers more carefully – Periodicity

@L.Guthardt, please, don't write such comments. They do not help people, while this answer does. @exiaohao, it seems, Docker uses different DNS (not 127.0.0.11) when Weave address space is constrained by CIDR. Setting the resolver to nameservers IP noted in containers /etc/resolv.conf helped. Thanks. – Jacquenette

Using the value in /etc/resolv.conf worked for AWS ECS Docker containers. – Veil

Answer 3

I was running "node:12.18-alpine" with angular frontend and hit the same problem with proxy_pass.

Locally it was working with:

resolver 127.0.0.11;

As simple as that! Just execute:

$ cat /etc/resolv.conf | grep nameserver

In your container to get this ip address.

However, when deploying to kubernetes (AWS EKS) I got the very same error:

failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53

Solution:

First solution was to find out the IP of the kube-dns service like below:

$ kubectl get service kube-dns -n kube-system
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   172.20.0.10   <none>        53/UDP,53/TCP   178d

Simple replacing IP for CLUSTER-IP worked like a charm.

Later, after some more doc digging, I find out that I could reference the service by name (which is little bit more elegant and resilient):

resolver kube-dns.kube-system valid=10s;

Doiron

If nginx manages to resolve "kube-dns.kube-system" then I don't get why this needs to exist in the configuration. However, this is the only thing that worked for me in k8s. I don't think k8s uses docker anymore so the 127.0.0.11 trick doesn't work anymore. – Doglike

@Phil, when you deploy to kubernetes /etc/resolv.conf has its nameserver changed to kube-dns IP. That's why it can resolve "kube-dns.kube-system". "127.0.0.11" is what I use to run it locally. Regarding docker and k8s support, well, I'm still on 1.18 :-) – Doiron

Yeah I see, but resolving a resolver feels a bit incestrious. If nginx can already resolve the resolver name using OS DNS/resolv.conf, why does it still need to be told how to resolve other names such as those in proxy_pass. – Doglike

I see your point now. True thing. Unfortunately, I cannot answer that, but just say that's what works for me too :-( – Doiron

Answer 4

My problem was $request_uri at the end. After adding it at the end of uri and changing the 127.0.0.1 to 127.0.0.11 solved my issue. I hope it will help people to not spend hours on this.

location /products {
            resolver 127.0.0.11;
            proxy_pass http://products:3000$request_uri;
            }

Lynelllynelle

Answer 5

In several cases where I had this error, adding resolver_timeout 1s; to the Nginx config solved the issue. Most of the time I don't have a resolver entry.

Edit: what also worked for containers where I could explicitly define a nameserver: resolver DNS-IP valid=1s;

Nutgall

Answer 6

We hit this with docker containers on windows trying to lookup host.docker.internal using the docker internal resolver at 127.0.0.11. All queries would resolve correctly except host.docker.internal. Fix was to add the ipv6=off flag to the resolver line in nginx.conf.

Encapsulate

Did you mean 127.0.0.11, not 172.0.0.11? – Carte

I did. Corrected. Sorry! – Encapsulate

Answer 7

I solved this problem with the following way:

docker run --rm -d --network host --name "my_domain" nginx

https://docs.docker.com/network/network-tutorial-host/

Rattat

Answer 8

On Linux, the accepted answer work just fine for connecting containers under the same Docker network but it'll not work for host.docker.internal or any domain using --add-host or extra_hosts in compose because Docker's default DNS 127.0.0.11 does not include extra_hosts and --add-host entries which are stored to /etc/hosts file. 2 simple workarounds are:

1. Change host.docker.internal to 172.17.0.1 which it's not sure whether Docker will always use the same IP.

2. Install dnsmasq and run dnsmasq inside the container. Set nginx resolver to 127.0.0.1.

Loading

Answer 9

You need a local dns server like dnsmasq to resolve using 127.0.0.1. Try installing it using apk add --update dnsmasq and set it up if you're using an alpine (nginx:alpine) variant.

Fingertip