What is the proper way to start dnsmasq in my docker ENTRYPOINT?

Question

I wrote a simple script in my docker ENTRYPOINT to use dnsmasq

if [ ! -f /etc/resolv.dnsmasq ];then
   cp /etc/resolv.conf /etc/resolv.dnsmasq
   sed -i 's/^nameserver.*/nameserver 127.0.0.1/' /etc/resolv.conf
   dnsmasq -r /etc/resolv.dnsmasq
else
   dnsmasq -r /etc/resolv.dnsmasq
fi

My logic is simple, the first time k8s starts my docker contain, it will update nameserver in /etc/resolv.conf to the correct value and my script will copy it to /etc/resolv.dnsmasq and change nameserver in /etc/resolv.conf to 127.0.0.1 to use dnsmasq service.

When docker restarts because /etc/resolv.dnsmasq remains so dnsmasq will just start. It works most of time but when k8s restarts container due to health check failure it will fail to work. When that happen my resolv.dnsmasq has nameserver 127.0.0.1,

/var/www/html # cat /etc/resolv.dnsmasq
nameserver 127.0.0.1
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:2 timeout:2

From the comment and answer I got so far I realize it was because when the container was recreated then I have a modified /etc/resolv.conf and no /etc/resolv.dnsmasq. so the then part is run.

But why will I have nameserver 127.0.0.1 in /etc/resolv.conf if it is recreated from the image ?

How can I deal with that and make dnsmasq work ?

--- update---

Through our investigation it seems that sometimes Kubernetes will kill the running docker container and start a new container from image, but with the pause container exists the modified /etc/resolve.conf also exists. When that happens the restarted container does not have /etc/resolv.dnsmasq but it shares the /etc/resolv.conf that was already modified nameserver 127.0.0.1

I am not sure the whole logic behind the pause container. What is the use of a pause image in Kubernetes? said "The pause container holds the network namespace for the pod." So does that mean /etc/resolv.conf remains in my case? Can someone help to answer it ?

Nabokov

It's not that the file is deleted. Files that are created inside the container continue to exist unless the container needs to be recreated from the image. If it needs to be recreated, only the files that exist on the image will exist. – Doxia

Hi I saw you removed my second question. So I plan to rewrite my question to ask it because that is the main point of my question. And according to you it is because container is recreated. Is that okay with you ? – Nabokov

Hi I have updated my question. But why will I have a modified /etc/resolv.conf if it is recreated from the image ? – Nabokov

Have you considered using postStart hook in your Deployment? You would move your ENTRYPOINT script there. It should fix your problem in k8s cluster but it requires to change Dockerfile. Let me know what you think and if you are planning to use this container standalone or only in cluster. – Abie

In Kubernetes, do you actually need to run dnsmasq? Kubernetes has its own DNS system and a way to define additional names in it. – Scorpius

That is another question. I had thought I don't need dnsmasq either but it turns out that in some Kubernetes environment, the DNS resolve keeps failed and when I introduced dnsmasq the problem was solved! Check my question here #67529542 – Nabokov

The biggest downside of Alpine-based images is that Alpine has a different implementation of the core C library and sometimes there are compatibility issues. This sounds like an outright bug more than a compatibility issue, but you might try rebuilding your image against something based on the more standard GNU libc (typically a Debian or Ubuntu base) and see if that helps. – Scorpius

@Qiulang邱朗 have you considered my suggestion? – Abie

@Abie Not yet. I need to fix this particular question first. – Nabokov

Can you provide your Dockerfile? – Abie

I still hit DNS failure intermittently without dnsmasq setup correctly and I find this article explains the reason "musl (by design) doesn't support DNS-over-TCP... when a a single UDP packet (512 bytes) is not enough to resolve hostnames" – Nabokov

Answer 1

...when k8s restarts container due to health check failure it always fail to work

Different from docker restart, K8s restart container with new instance, anything you saved in the killed container will not be there. For your file(s) to persist thru K8s restart, you need to save them in persistent volume (eg. NFS, storage supplied by your cloud provider etc).

Aklog

Hi I updated my question, can you take a look again ? – Nabokov

Checkout the updated answer. – Quantifier

But why will I have a modified /etc/resolv.conf if it is recreated from the image ? – Nabokov

That is exactly my question! I don't think /etc/resolv.conf has 127.0.0.1 in image. But is it ? I am not 100% sure now. – Nabokov

Not sure what you mean, note by default a pod inherits the name resolution configuration from the node. – Quantifier

Now my /etc/resolv.dnsmasq has nameserver 127.0.0.1. I don't know how that could happen. – Nabokov

Can you post the first version of /etc/resolv.conf before you copy into another file? Note all comments made here is for k8s and not docker. – Quantifier

I can if ! grep -q "127.0.0.1" /etc/resolv.conf && [ ! -f /etc/resolv.dnsmasq ]; then ... but that won't solve my problem, i.e. to use dnsmasq – Nabokov

Can you do a cat /etc/resolv.conf before your bash script, then use kubectl logs <pod> to see the original content of resolv.conf. – Quantifier

By the way, the cat is to be run during pod start up like initContainers, not in your Dockerfile ENTRYPOINT/CMD or docker-entrypoint.sh. – Quantifier

Can you check my update ? – Nabokov