Why dig does not resolve K8s service by dns name while nslookup has no problems with it?

Question

Here are steps to reproduce:

minikube start
kubectl run nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=ClusterIP

kubectl run -i --tty --rm alpine --image=alpine --restart=Never -- sh
apk add --no-cache bind-tools

Now let's try to query kibe-dns for nginx service

with nslookup:

/ # nslookup nginx.default 10.96.0.10
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   nginx.default.svc.cluster.local
Address: 10.97.239.175

and with dig:

dig nginx.default @10.96.0.10 any

; <<>> DiG 9.11.3 <<>> nginx.default @10.96.0.10 any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46414
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nginx.default.         IN  ANY

;; Query time: 279 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Jun 03 15:31:15 UTC 2018
;; MSG SIZE  rcvd: 42

Nothing changes if I replace name nginx.default with just nginx.

minikube version: v0.27.0, k8s version: 1.10.0

Rutilant

Answer 1

Answer

Dig does not complete the query by default with the search path. The search path is set in /etc/resolv.conf. The +search flag enables the search path completion.

From the Man Pages

+[no]search
Use [do not use] the search list defined by the searchlist or domain directive in resolv.conf (if any). The search list is not used by default.

https://linux.die.net/man/1/dig

Demonstration

I have created a scenario for katacoda which goes through the same example interactively https://www.katacoda.com/bluebrown/scenarios/kubernetes-dns

First create and expose a pod, then start another pod interactively with dnsutils installed, from which DNS queries can be made.

kubectl create namespace dev
kubectl run my-app --image nginx --namespace dev --port 80
kubectl expose pod my-app --namespace dev
kubectl run dnsutils --namespace dev --image=bluebrown/netutils --rm -ti

Nslookup resolves the service OK

$ nslookup my-app
...
Name:   my-app.dev.svc.cluster.local
Address: 10.43.52.98

But dig didn't get an answer, why?

$ dig my-app
...
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
...

In order to understand why dig doesn't find the service, let's take a look at /etc/resolv.conf

$ cat /etc/resolv.conf
search dev.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

This file contains a line with the following format.

search <namespace>.svc.cluster.local svc.cluster.local cluster.local

That means, when providing an incomplete part of the fully qualified domain name (FQDN), this file can be used to complete the query. However, dig doesn't do it by default. We can use the +search flag in order to enable it.

dig +search my-app
...
;; QUESTION SECTION:
;my-app.dev.svc.cluster.local.  IN      A

;; ANSWER SECTION:
my-app.dev.svc.cluster.local. 5 IN      A       10.43.52.98

Now the service-name has been correctly resolved. You can also see how the query has been completed with the search path by comparing the question section of this command with the previous one without +search flag.

We can get the same service without +search flag when using the FQDN. The +short flag isn't required, but it will reduce the output to only the IP address.

$ dig +short my-app.dev.svc.cluster.local
10.43.52.98

However, the benefit of using the search method it that queries will automatically resolve to resources within the same namespace. This can be useful to apply the same configuration to different environments, such as production and development.

The same way the search entry in resolv.conf completes the query with the default name space, it will complete any part of the FQDN from left to right. So in the below example, it will resolve to the local cluster.

$ dig +short +search my-app.dev
10.43.52.98

Rat

Answer 2

Few things to point out:

1. Always use "nginx.default.svc.cluster.local", and don't rely on the tool to complete it for you.
2. Run dig nginx.default.svc.cluster.local NS you will see there are no hosting nameserver for it. You can only specify one when there are at least one exists.
3. Interestingly I found out that dig any in alpine doesn't really work so, so in this case, I am afraid that you have to explicitly use A or left it empty(A is default)

Conclusion:

Run dig nginx.default.svc.cluster.local or dig nginx.default.svc.cluster.local A instead.

Mikael

Thanks. Both options proposed give me DNS resolution with dig. The question actually appeared after some experience with HaProxy. I have a resolvers section and backend server with dns name instead of network address. I tried to check if HaProxy continues proxying when dns name first stops resolving and continues resolving in a while. The fun thing is that nginx is correctly resolved on startup. But it fails to check server backend by this "short" name every 30 seconds. HaProxy resolver also prefers nginx.default.svc.cluster.local even though it can handle just nginx on startup. – Rutilant

dig does not use search paths by default. Try dig +search shortname. Reference: github.com/kubernetes/kubernetes/issues/16744 – Johore