How to get into CoreDNS pod kuberrnetes?

Question

I have a running k8s cluster with two replicas of CoreDNS. But when i try enter the bash prompt of the POD it's throwing me below error

# kubectl exec -it coredns-5644d7b6d9-285bj -n kube-system sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "94f45da89fa5493a8283888464623788ef5e832dc31e0d89e427e71d86391fd6": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown

But i am able to login to other pods without any issues. I tried with nsenter with kernel process ID it works but it only works for network related openrations like,

# nsenter -t 24931 -n ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
    link/ether 7a:70:99:aa:53:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.2/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::7870:99ff:feaa:536c/64 scope link
       valid_lft forever preferred_lft forever

How to enter into this pod using kubectl and get rid of that error?

Goer

why do you wanna get into CoreDNS? what is it you are trying to debug? – Monad

I am trying to check the entry in /etc/coredns/Corefile for one of my application service. – Goer

Answer 1

You can use the sidecar pattern following the instructions here: https://support.rancher.com/hc/en-us/articles/360041568712-How-to-troubleshoot-using-the-namespace-of-a-container#sidecar-container-0-2

In short, do this to find a node where a coredns pod is running:

kubectl -n kube-system get po -o wide | grep coredns

ssh to one of those nodes, then:

docker ps -a | grep coredns

Copy the Container ID to clipboard and run:

ID=<paste ID here>
docker run -it --net=container:$ID --pid=container:$ID --volumes-from=$ID alpine sh

You will now be inside the "sidecar" container and can poke around. I.e.

cat /etc/coredns/Corefile

Bicuspid

My K8s is using containerd and docker does not show any containers, but crictl does. However, crictl does not understand all of those command line parameters. Is there a wariant of this fpr crictl? – Guffey

Answer 2

There is a way of getting access to the filesystem of the coredns pod in Kubernetes.

Debugging with ephemeral containers is the way to go as the image does not contain any shell.

$ kubectl debug -it coredns-6d4b75cb6d-77d86 --image=busybox:1.28 --target=coredns

I changed to kube-system namespace using

$ kubectl config set-context --current --namespace=kube-system

But the -n option can also be used in the command.

Attaching a ephemeral container with --target option enables process namespace sharing

After getting access to the terminal, you can view processes with:

$ ps aux
PID   USER     TIME  COMMAND
    1 root      0:08 /coredns -conf /etc/coredns/Corefile
  210 root      0:00 sh
  266 root      0:00 ps aux

The PID of the coredns process is 1 and the container filesystems are visible to other containers in the pod through the /proc/$pid/root link. This makes debugging easier, but it also means that filesystem secrets are protected only by filesystem permissions. (docs)

Finally, the config file can be viewed at

$ cat /proc/1/root/etc/coredns/Corefile
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Deane

Can you safely edit and create files, or should you only inspect them? – Lipase

As a general rule of thumb, don't modify something that was set up by Kubernetes controllers without using the Kubernetes API. So in short, don't modify only inspect. – Deane

Answer 3

If you are trying to check the Corefile then you can run below

kubectl get cm coredns -n kube-system -o yaml

Monad

But this will be the intial configuration right? Once the PODs are added, coredns will start adding the domains of each pods into dns table right? How can i check that? – Goer

That will not be there in core dns pod. You need to check in etc/resolve.conf file inside the application pod. BTW each pod don't have their own domain – Monad

Ok, so when i do nslookup (from busybox for suppose) from some it reaches the DNS ip and resolves it right? Where does it tries to lookup and resolve it? – Goer

/etc/resolve.conf will have core dns pod ip as nameserver – Monad