how does Docker Embedded DNS resolver work?

Question

I know Docker has a embedded Dns resolver.
when I run a container in myself bridge:

$ docker run -it --rm --privileged --network=mybridge xxx bash

root@18243bfe6b50:/# cat /etc/resolv.conf  
nameserver 127.0.0.11  
options ndots:0  

root@18243bfe6b50:/# netstat -anop  
Active Internet connections (servers and established)  
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name Timer  
tcp        0      0 127.0.0.11:45997        0.0.0.0:*               LISTEN      -                off (0.00/0/0)  
udp        0      0 127.0.0.11:49614        0.0.0.0:*        

it shows there is a dns resolver, and iptables help do a port transfer.  

root@18243bfe6b50:/# iptables -nvL -t nat  
.....  
Chain DOCKER_OUTPUT (1 references)  
 pkts bytes target     prot opt in     out     source               destination  
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:45997  
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:49614  

Chain DOCKER_POSTROUTING (1 references)  
 pkts bytes target     prot opt in     out     source               destination  
    0     0 SNAT       tcp  --  *      *       127.0.0.11           0.0.0.0/0            tcp spt:45997 to::53  
    0     0 SNAT       udp  --  *      *       127.0.0.11           0.0.0.0/0            udp spt:49614 to::53  

but, which process is the dns resolver? I guess it is dockerd?  but dockerd is running in host network namespace, obviously it is different with the container network namespace, also, I can not find dockerd has dns port listening in host:  

root@test:~# netstat -tnop |grep dockerd  
tcp        0      0 10.5.79.50:59540        10.5.79.50:2377         ESTABLISHED 3332/dockerd     off (0.00/0/0)  
tcp        0      0 127.0.0.1:35792         127.0.0.1:2377          ESTABLISHED 3332/dockerd     off (0.00/0/0)  
tcp6       0      0 10.5.79.50:2377         10.5.79.70:45934        ESTABLISHED 3332/dockerd     off (0.00/0/0)  
tcp6       0      0 127.0.0.1:2377          127.0.0.1:35792         ESTABLISHED 3332/dockerd     off (0.00/0/0)  
tcp6       0      0 10.5.79.50:2377         10.5.79.50:59540        ESTABLISHED 3332/dockerd     off (0.00/0/0)  

how does one process(dockerd) expose some ports in host namespace and some ports in other namespace(container)? I read some code, but still can not figure out, could anyone help answer?

thanks.

Guerrero

To add to the Great answer by @gesellix, you can get deeper understanding from here – Amand

use “iptables-legacy” if you can't see the DNAT and SNAT rules. because the iptables may be a link to iptables-nft , like the image: Ubuntu 22.04.1 LTS – Bismuthinite

Answer 1

Maybe you have already found that Docker (aka Moby) internally uses libnetwork to configure and enable the embedded DNS resolver. Libnetwork binds the resolver to the container's loopback interface, so that DNS queries at 127.0.0.11 can be routed (via iptables) to the "backend DNS resolver" in the Docker Engine. See the libnetwork type and the actual ResolveName() code. The Sandbox for each container allows to route DNS queries through the network namespaces.

Regarding your question how one process can expose ports on the host and inside a container: in this case binding a handler thread to an interface would be a more appropriate expression. The Docker engine creates a container and configures its network namespace, so it can also configure the container's network interfaces and packet routing via iptables. Binding the resolver to the container's internal interface is the reason why you haven't found any process on the host listening on port 53.

Adlai

nice answer (you probably meant packet routing, instead of package routing). – Tamarra

Answer 2

Thanks @gesellix

What Docker do：

1. The embedded DNS resolver (dockerd) switch net NS to the Container (use setns system call)
2. listen at 127.0.0.11:xxxx, and return to its Origin net NS.
3. use iptables DNAT and SNAT rule, forwarding port 53 to xxxxx.
4. use /etc/resolv.conf to nameserver 127.0.0.11

---

I guess because the dockerd not in pid_namespace of the Container, So you can't see PID/Program name in the Container.

To see dockerd listening on 127.0.0.11:

1. get container ID: docker ps
2. get container PID: docker inspect --format "{{.State.Pid}}" 89c249fc51f6
3. run command in container net NS: nsenter -n -t 3794 ss -lnutp (alternatively: nsenter -n -t 3794 netstat -anop)

you will see something like:

    Netid State   Recv-Q  Send-Q    Local Address:Port    Peer Address:Port                                                                                       
    udp   UNCONN  3072    0            127.0.0.11:43418        0.0.0.0:*      users:(("dockerd",pid=1348,fd=63))
    tcp   LISTEN  0       128          127.0.0.11:34643        0.0.0.0:*      users:(("dockerd",pid=1348,fd=65))

4. to see iptables rules: nsenter -n -t 3794 iptables -nvL -t nat

(In some Linux release, use iptables-legacy instead of iptables)

Chain DOCKER_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            127.0.0.11           tcp dpt:53 to:127.0.0.11:34643
    4   242 DNAT       udp  --  *      *       0.0.0.0/0            127.0.0.11           udp dpt:53 to:127.0.0.11:43418

Chain DOCKER_POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 SNAT       tcp  --  *      *       127.0.0.11           0.0.0.0/0            tcp spt:34643 to::53
    0     0 SNAT       udp  --  *      *       127.0.0.11           0.0.0.0/0            udp spt:43418 to::53

---

About the libnetwork code：

In Resolver

    // SetupFunc() provides the setup function that should be run
    // in the container's network namespace.

Call InvokeFunc here

if err := sb.osSbox.InvokeFunc(sb.resolver.SetupFunc(0));

def InvokeFunc

func (n *networkNamespace) InvokeFunc(f func()) error {
    return nsInvoke(n.nsPath(), func(nsFD int) error { return nil }, func(callerFD int) error {
        f()
        return nil
    })
}

In nsInvoke you will see netns.Set

...
netns.GetFromPath(path)
...
netns.Set(newNs)
...

Go https://github.com/vishvananda/netns to see the details of the function

Bismuthinite