Kubernetes kube-dns pod is pending

Question

I tried this doc to install and setup Kubernetes in Ubuntu VM. I have finished upto 3/4 and now kube-dns pod is in pending status. How can i figure out this? here is the result for kubectl get pods --namespace=kube-system and kubectl describe pod <pod name>

# kubectl get pods --namespace=kube-system
NAME                              READY     STATUS    RESTARTS   AGE
dummy-2088944543-jk2t2            1/1       Running   0          3h
etcd-ubuntu                       1/1       Running   0          3h
kube-apiserver-ubuntu             1/1       Running   0          3h
kube-controller-manager-ubuntu    1/1       Running   0          3h
kube-discovery-1769846148-h88v4   1/1       Running   0          3h
kube-dns-2924299975-dfp17         0/4       Pending   0          3h
kube-proxy-zdcxw                  1/1       Running   0          3h
kube-scheduler-ubuntu             1/1       Running   0          3h
weave-net-xwfhj                   2/2       Running   0          2h

# kubectl describe pod kube-dns-2924299975-dfp17
Error from server (NotFound): pods "kube-dns-2924299975-dfp17" not found

Derna

Try using kubectl command to inspect the labels in the configuration data of the pod. Kubernetes well only schedule it only a host/kubelet which has matching labels. It could be that the pod for example has labels stating to only run in on a node which has a label "region=infrastructure" or something similar (that's just a made up example).also each kubelet has a notion of whether it is "schedulable" and it won't get anything automatically deployed onto it if this is not turned on. The last idea is that it possibly cannot download the container image. You can check the logs for that – Farmstead

@lakmal-vithanage can you post the output of kubectl describe pod <pod-name>? – Ollayos

@AntoineCotten, I have updated the question – Derna

@lakmal-vithanage you forgot --namespace=kube-system in the command, hence the error. Could you update the question one more time? – Ollayos

Answer 1

Cause

Most likely a lack of available computing resources in your cluster.

If you're using the example in cluster/addons/dns you're certainly using a Deployment with resource requests, highlighted if you click the link. It could be that your other pods are already requesting all the available resources in the cluster, therefore your pod doesn't get scheduled.

You can confirm that theory with kubectl --namespace=kube-system describe pod kube-dns-2924299975-dfp17 and look for the following event:

Reason                Message
------                -------
FailedScheduling      pod (kube-dns-2924299975-dfp17) failed to fit in any node
fit failure summary on nodes : Insufficient cpu (3)

You can also describe your nodes with kubectl describe node <node-name> and look at the last information:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits      Memory Requests Memory Limits
  ------------  ----------      --------------- -------------
  320m (8%)     300m (7%)       150Mi (1%)      150Mi (1%)

In your case either the CPU or memory allocation should be close to 100%.

Solution

• Add more computing resources / nodes to your cluster (preferred)
• Remove the resource requests from your pod(s), at the risk of overcommitting your resources

Ollayos

this was also the issue in my case. Upgraded masters from t2.micro to t2.small and that fixed it. – Jaehne

@Jaehne Masters should not be schedulable in general, for security reasons. Upgrading them so that you can schedule pods on them is quite an anti-pattern. – Ollayos

yes, good point. In my case it was the kube-scheduler that couldn't be deployed... ; ) – Jaehne

Is it possible to decrease/remove kube-dns default requests? – Kimsey