Pods are not starting. NetworkPlugin cni failed to set up pod

Question

K8 Version:

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

I tried to launch spinnaker pods(yaml files here). I choose Flannel(kubectl apply -f kube-flannel.yml) while installing K8. Then I see the pods are not starting, it is struck in "ContainerCreating" status. I kubectl describe a pod, showing NetworkPlugin cni failed to set up pod

veeru@ubuntu:/opt/spinnaker/experimental/kubernetes/simple$ kubectl describe pod data-redis-master-v000-38j80 --namespace=spinnaker
Name:       data-redis-master-v000-38j80
Namespace:  spinnaker
Node:       ubuntu/192.168.6.136
Start Time: Thu, 01 Jun 2017 02:54:14 -0700
Labels:     load-balancer-data-redis-server=true
        replication-controller=data-redis-master-v000
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"spinnaker","name":"data-redis-master-v000","uid":"43d4a44c-46b0-11e7-b0e1-000c29b...
Status:     Pending
IP:     
Controllers:    ReplicaSet/data-redis-master-v000
Containers:
  redis-master:
    Container ID:   
    Image:      gcr.io/kubernetes-spinnaker/redis-cluster:v2
    Image ID:       
    Port:       6379/TCP
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Limits:
      cpu:  100m
    Requests:
      cpu:  100m
    Environment:
      MASTER:   true
    Mounts:
      /redis-master-data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-71p4q (ro)
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  data:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-71p4q:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-71p4q
    Optional:   false
QoS Class:  Burstable
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  45m       45m     1   default-scheduler           Normal      Scheduled   Successfully assigned data-redis-master-v000-38j80 to ubuntu
  43m       43m     1   kubelet, ubuntu             Warning     FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: 8265d80732e7b73ebf8f1493d40403021064b61436c4c559b41330e7592fd47f"

  43m   43m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: b972862d763e621e026728073deb9a304748c4ec4522982db0a168663ab59d36
  42m   42m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: 72b39083a3a81c0da1d4b7fa65b5d6450b62a3562a05452c27b185bc33197327"

  41m   41m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: d315511bfa9f6f09d7ef4cd277bde44e4885291ea566e3089460356c1ed34413"

  40m   40m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: a03d776d2d7c5c4ae9c1ec31681b0b6e40759326a452916cff0e60c4d4e2c954"

  40m   40m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: acf30a4aacda0c53bdbb8bc2d416704720bd1b623c43874052b4029f15950052"

  39m   39m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: ea49f5f9428d585be7138f4ebce54f713eef549b16104a3d7aa728175b6ebc2a"

  38m   38m 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "447d302c-46b0-11e7-b0e1-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: ec2483435b4b22576c9bd7bffac5d67d53893c189c0cf26aca1ae6af79d09914"

  38m   1m  39  kubelet, ubuntu     Warning FailedSync  (events with common reason combined)
  45m   1s  448 kubelet, ubuntu     Normal  SandboxChanged  Pod sandbox changed, it will be killed and re-created.
  45m   0s  412 kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "CreatePodSandbox" for "data-redis-master-v000-38j80_spinnaker(447d302c-46b0-11e7-b0e1-000c29b1270f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"data-redis-master-v000-38j80_spinnaker(447d302c-46b0-11e7-b0e1-000c29b1270f)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"data-redis-master-v000-38j80_spinnaker\" network: open /run/flannel/subnet.env: no such file or directory"

How can I resolve above issue?

UPDATE-1

I have reinitialized K8 with kubeadm init --pod-network-cidr=10.244.0.0/16 and deployed sample nginx pod. Still getting same error

-----------------OUTPUT REMOVED------------------------------- 
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  3m        3m      1   default-scheduler           Normal      Scheduled   Successfully assigned nginx-622qj to ubuntu
  1m        1m      1   kubelet, ubuntu             Warning     FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "0728fece-46fe-11e7-ae5d-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: 38250afd765f0108aeff6e31bbe5a642a60db99b97cbbf15711f810cbe8f3829"

  24s   24s 1   kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "0728fece-46fe-11e7-ae5d-000c29b1270f" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"_\" network: CNI failed to retrieve network namespace path: Error: No such container: 3bebcef02cb5f6645a65dcf06b2730144080f9d3c4fb18267feca5c5ce21031c"

  2m    9s  33  kubelet, ubuntu     Normal  SandboxChanged  Pod sandbox changed, it will be killed and re-created.
  3m    7s  32  kubelet, ubuntu     Warning FailedSync  Error syncing pod, skipping: failed to "CreatePodSandbox" for "nginx-622qj_default(0728fece-46fe-11e7-ae5d-000c29b1270f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-622qj_default(0728fece-46fe-11e7-ae5d-000c29b1270f)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"nginx-622qj_default\" network: open /run/flannel/subnet.env: no such file or directory"

Witenagemot

For flannel as cni the apiserver needs to have the argument --pod-network-cidr=... to be set to the overlay. Have you got that done? How did you start the cluster? – Kelt

After I ran kubeadm init, I have installed pod network with kubectl apply -f kube-flannel.yml. – Witenagemot

@Janos Lenart, please check my "UPDATE-1" in question. – Witenagemot

Answer 1

You error message shows flanel subnet.evn file is missing. you need to fix flannel configuration first. What version of kubernetes your using?

network: open /run/flannel/subnet.env: no such file or directory"

if your using kubernetes 1.6 and above, we can use below yaml file to configure the flannel container process.

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Diamagnet

Run on the master node. – Diamagnet

thanks for reply, so no need to create network service in worker nodes right ? – Oceanid

@Diamagnet The link is not accessible anymore. It returns a 404. Do you have an alternative link? – Reputed

Updated the link can you try now. – Diamagnet

Answer 2

Running the following command resolved my issues:

kubeadm init --pod-network-cidr=10.244.0.0/16

For flannel as cni the api server needs to have the argument --pod-network-cidr=... to be set to the overlay.

Lambdacism

Answer 3

Here are the steps that fixed my issue.

1. Create a file called subnet.env at location /run/flannel/ inside your worker nodes.

2. Add the below content in it.

   FLANNEL_NETWORK=10.244.0.0/16
   FLANNEL_SUBNET=10.244.0.1/24
   FLANNEL_MTU=1450
   FLANNEL_IPMASQ=true

3. Save the file and create the pod again. It should saw the runnign status now.

Excurrent

Answer 4

I was getting same error on one of node after disk pressure and some how it deleted file /run/flannel/subnet.env.

After creating file with content from other node resolve my issue.

Ampereturn