Pod unable to install packages(apt-get update or apt-get install )

Question

I have observed that the pods in my cluster is not able to install the packages when exec to the pod . While debugging i have realized that its due to /etc/resolv.conf entries.

The /etc/resolv.conf entry from one of the pod is :

nameserver 192.168.27.116
search ui-container.svc.cluster.local svc.cluster.local cluster.local 192.168.27.116.nip.io
options ndots:5

Here if i remove a entry 192.168.27.116.nip.io from the resolv.conf of all the master, worker nodes then the pods will be able to connect to internet and apt-get update and apt-get install works. This is only a temporary workaround because it is not recommended to update the resolv.conf. because i have observed that resolv.conf contents gets re-set to original upon reboot of the nodes.

Is it due to options ndots:5 in the /etc/resolv.conf ?

How can i fix this?

Nauseating

Answer 1

As a quick-fix, you could leverage dnsConfig from pod Spec to override default dns configuration, more details: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config

Myelencephalon

Answer 2

No, it's because of the nip.io address in your resolv.conf. It's a special domain that's used for reflection and mocking. It's not the ndots:5.

Fp

Thanks for the info. The entries in the /etc/resolv.conf are auto-generated. How can i solve the issue then? – Nauseating

how you installed cluster? – Recipient

Seems that the domain set for the cluster is the 192.168.27.116.nip.io. One option would be change the default domain for your cluster so it wouldn't use the nip.io. That would fix every pod you have in your cluster. You can also change the search per pod, using dnsConfig. – Fp

Yes cluster hostname is set to 192.168.27.116.nip.io since it eliminated our need to edit the /etc/hosts entry for the routes. Thanks for your suggestions for fix. Im looking for alternate solution other any removing nip.io or setting search for pod. – Nauseating

@Vitalii the cluster was installed using github.com/openshift/openshift-ansible/tree/release-3.11 – Nauseating

The issue is that nip.io is a wide ranging domain. Because of that, anything that you throw at them will be forwarded to 192.168.27.116. If you can override the cluster settings I guess you could mess around with the order domains are resolved (such as including top domains before your 192.168.27.116.nip.io), but that's a long shot and could potentially leak information from your cluster and/or slow things down. Good luck! – Fp

What I don't understand is why the "search" needs to be used, if the URL for apt server is correct? What I understand the search is options that will be appended to the hostname just in the case that the hostname cannot be resolved. – Mashe

You're getting it right @ChayneP.S.. The problem here is that the resolv.conf is appending for long domains as well, because of the ndots:5 option. That's a configuration set for the search to be relayed straight to public DNS when there's more than 5 dots in the domain, and setting it lower or higher could be problematic because it could mess with the cluster internal name resolution. – Fp

@GustavoKawamoto Thank you for explaining. It's clear to me now. – Mashe

Answer 3

Adding to above answers, usually resolve.conf follows following pattern

nameserver <local domain server provided by docker>
search <pod namespace>.svc.cluster.local svc.cluster.local cluster.local <host/node's actual DNS>
options ndots:5

Last part of the DNS <host/node's actual DNS> is usually your LAB's or system wide DNS that is part of your network. So you may have override with DNS policy or update the same in your lab. nip.io is a FQDN maker where there is a need, so you may come up with a local DNS name that resolves within your LAB instead of relying on that.

Heptane