kubectl get nodes shows NotReady [closed]

Question

I have installed two nodes kubernetes 1.12.1 in cloud VMs, both behind internet proxy. Each VMs have floating IPs associated to connect over SSH, kube-01 is a master and kube-02 is a node. Executed export:

no_proxy=127.0.0.1,localhost,10.157.255.185,192.168.0.153,kube-02,192.168.0.25,kube-01

before running kubeadm init, but I am getting the following status for kubectl get nodes:

NAME      STATUS     ROLES    AGE   VERSION
kube-01   NotReady   master   89m   v1.12.1
kube-02   NotReady   <none>   29s   v1.12.2

Am I missing any configuration? Do I need to add 192.168.0.153 and 192.168.0.25 in respective VM's /etc/hosts?

Identify

can u please paste the output of "kubect get pods -n kube-system" – Urbain

NAME READY STATUS RESTARTS AGE coredns-576cbf47c7-krtb6 0/1 Pending 0 54m coredns-576cbf47c7-nvgg8 0/1 Pending 0 54m etcd-kube-01 1/1 Running 0 53m kube-apiserver-kube-01 1/1 Running 0 53m kube-controller-manager-kube-01 1/1 Running 0 53m kube-proxy-5rxfq 1/1 Running 0 54m kube-scheduler-kube-01 1/1 Running 0 53m – Identify

have you installed any CNI ? like flannel or weave ? – Urbain

install a cni on your master and check. to apply flannel use " kubectl apply -f raw.githubusercontent.com/coreos/flannel/master/Documentation/… " this should work – Urbain

Answer 1

Looks like pod network is not installed yet on your cluster . You can install weave for example with below command

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

After a few seconds, a Weave Net pod should be running on each Node and any further pods you create will be automatically attached to the Weave network.

You can install pod networks of your choice . Here is a list

after this check

$ kubectl describe nodes

check all is fine like below

Conditions:
  Type              Status
  ----              ------
  OutOfDisk         False
  MemoryPressure    False
  DiskPressure      False
  Ready             True
Capacity:
 cpu:       2
 memory:    2052588Ki
 pods:      110
Allocatable:
 cpu:       2
 memory:    1950188Ki
 pods:      110

next ssh to the pod which is not ready and observe kubelet logs. Most likely errors can be of certificates and authentication.

You can also use journalctl on systemd to check kubelet errors.

$ journalctl -u kubelet

Urbain

my worker node still shows Not ready root@kube-01:~# kubectl get nodes NAME STATUS ROLES AGE VERSION kube-01 Ready master 63m v1.12.1 kube-02 NotReady <none> 51m v1.12.2 – Identify

what is the result of kubect get pods -n kube-system and "kubectl describe node <worker node name>" – Urbain

getting following The connection to the server localhost:8080 was refused - did you specify the right host or port? – Identify

you need to run kubect get pods -n kube-system and "kubectl describe node <worker node name>" on your master node. Kubectl cannot be run on worker node. Paste the results of those commands – Urbain

getting following error while joining node to cluster – Identify

what is the command you are using to join the node to master. You need to run that command as root on your worker node, make sure – Urbain

I am running from root user but getting following error – Identify

Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [preflight] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-02" as an annotation error uploading crisocket: timed out waiting for the condition – Identify

sudo cp /etc/kubernetes/admin.conf $HOME/ sudo chown $(id -u):$(id -g) $HOME/admin.conf export KUBECONFIG=$HOME/admin.conf did you run the above commands on master with regular user ? – Urbain

no there is no admin.conf following are files in /etc/kubernetes -rw------- 1 root root 1783 Oct 30 19:30 bootstrap-kubelet.conf -rw------- 1 root root 1853 Oct 29 18:37 kubelet.conf drwxr-xr-x 2 root root 4096 Oct 29 12:33 manifests drwxr-xr-x 2 root root 4096 Oct 29 18:37 pki – Identify

getting error [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [preflight] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-02" as an annotation error uploading crisocket: timed out waiting for the condition – Identify

Can you please share simple steps to install kubernetes v1.12.2 in one master one worker node – Identify

follow these steps on this link edureka.co/blog/install-kubernetes-on-ubuntu – Urbain

Answer 2

Try with this

Your coredns is in pending state check with the networking plugin you have used and check the proper addons are added

check kubernates troubleshooting guide

https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-or-kube-dns-is-stuck-in-the-pending-state

https://kubernetes.io/docs/concepts/cluster-administration/addons/

And install the following with those

And check

kubectl get pods -n kube-system

Incompressible

Answer 3

On the off chance it might be the same for someone else, in my case, I was using the wrong AMI image to create the nodegroup.

Latterll

Answer 4

Run

journalctl -u kubelet

Then check at node logs, if you get below error, disable the sawp using swapoff -a

"Failed to run kubelet" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fa Main process exited, code=exited, status=1/FAILURE

Truckload

Answer 5

I found that if you delete a node and re-provision and re-join it with the same name, WITHOUT issuing a node delete command, the node will join but report a NotReady state without much else to indicate the problem.

Likely an authentication issue bound to the previous system of the same name.

Either rename the new node or issue kubectl delete node <nodename>

Backslide