How to fix Flannel CNI plugin. Error: [plugin flannel does not support config version ""]

Question

While installing Kubernetes with, I'm stuck at CNI plugin installation and configuration part. I have installed Flannel but I see error in kubelet logs due to which coredns pods are in pending state.

OS: Centos7 k8s version: 1.16 Kubeadm is being used to setup the cluster.

I had installed the plugin using: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

This is error I can see in Kubelet logs:

Sep 21 04:47:29 peteelizalde2c kubelet: W0921 04:47:29.897759   17817 cni.go:202] Error validating CNI config &{cbr0  false [0xc000fb3ee0 0xc000fb3f60] [123 10 32 32 34 110 97 109 101 34 58 32 34 99 98 114 48 34 44 10 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 102 108 97 110 110 101 108 34 44 10 32 32 32 32 32 32 34 100 101 108 101 103 97 116 101 34 58 32 123 10 32 32 32 32 32 32 32 32 34 104 97 105 114 112 105 110 77 111 100 101 34 58 32 116 114 117 101 44 10 32 32 32 32 32 32 32 32 34 105 115 68 101 102 97 117 108 116 71 97 116 101 119 97 121 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 123 10 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 112 111 114 116 109 97 112 34 44 10 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 10 32 32 32 32 32 32 32 32 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 32 116 114 117 101 10 32 32 32 32 32 32 125 10 32 32 32 32 125 10 32 32 93 10 125 10]}: [plugin flannel does not support config version ""]
Sep 21 04:47:29 peteelizalde2c kubelet: W0921 04:47:29.897824   17817 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 21 04:47:32 peteelizalde2c kubelet: E0921 04:47:32.007379   17817 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Here is the pods:

kubectl get pods --all-namespaces
NAMESPACE     NAME                                                                        READY   STATUS    RESTARTS   AGE
kube-system   coredns-5644d7b6d9-n4h5x                                                    0/1     Pending   0          114m
kube-system   coredns-5644d7b6d9-t2q54                                                    0/1     Pending   0          114m
kube-system   etcd-ip-10-29-89-124                      1/1     Running   0          113m
kube-system   kube-apiserver-ip-10-29-89-124            1/1     Running   0          113m
kube-system   kube-controller-manager-ip-10-29-89-124   1/1     Running   0          113m
kube-system   kube-flannel-ds-amd64-dqpzj                                                 1/1     Running   0          110m
kube-system   kube-proxy-vzlqb                                                            1/1     Running   0          114m
kube-system   kube-scheduler-ip-10-29-89-124            1/1     Running   0          113m

There is a file in /etc/cni/net.d named 10-flannel.conflist. Its contents are:

{
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

Mcclintock

Answer 1

The accepted solution is incomplete and will cause headache down the line.

The proper solution to make this change permanent is to edit the ConfigMap created by flannel in your Kubernetes cluster. Otherwise, the file will be recreated the next time the flannel pod volumes are populated with the ConfigMap (e.g. on node reboot).

Use kubectl edit cm -n kube-system kube-flannel-cfg to edit the ConfigMap provided by flannel, and add the missing line:

  5 apiVersion: v1
  6 data:
  7   cni-conf.json: |
  8     {
  9       "name": "cbr0",
 10       "cniVersion": "0.2.0",
 11       "plugins": [

Reboot the node, or alternatively make the change manually in /etc/cni/net.d/10-flannel.conflist and do systemctl restart kubelet afterwards to skip the reboot.

Libove

Answer 2

The file /etc/cni/net.d/10-flannel.conflist was missing cniVersion key in its config.

Adding "cniVersion": "0.2.0" solved the issue.

Mcclintock

Confirm this solved my problem although I'm not too sure whether version 0.2.0 is the right version for me. How did you come up with that version? – Photodynamics

Confirmed also. – Vomer

Confirmed this fixed my issue as well. Kubernetes 1.16 Ubuntu 18 – Abecedary

This solved the error message but coredns pod is going to error and crashing with error as, [FATAL] plugin/loop: Loop (127.0.0.1:44415 -> :53) detected for zone "." – Headon

with Ubuntu and above changes codedns went into loop which was due to nameserver 127.0.1.1 in /etc/resolv.conf. By avoiding it using steps at hiroom2.com/2017/08/24/ubuntu-1610-nameserver-127-0-0-1-en it worked. CoreDNS is now running – Headon

Please see my solution. This will break the next time the node is rebooted, as the file is populated from a Kubernetes ConfigMap. – Bemba

Yes @Bemba this solution is a hotfix, your solution is a long term fix. – Mcclintock

Answer 3

Thanks this helped me as it was a show-stopper

https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf for the exact location of the "cniVersion": "0.2.0"

Screens

not sure if it is just me. But after I rebooted the master server the directive "cniVersion": "0.2.0” was again missing and I had to update the file again – Screens

See the accepted answer. The configmap is what should be edited. medium.com/@yonahdissen/… – Leandra