How to make the pod CIDR range larger in kubernetes cluster deployed with kubeadm?

Question

I deployed my cluster with the --pod-network-cidr added, and have created the new ip pool using calicoctl to change the pods to this range. The problem I am having is exactly what I need to change on the kubernetes side to make the pod cidr range changes? Do I make changes in the API server, Controller manager, and scheduler or is there only specific parts I need to change. I have attempted only changing the controller manager, and those control plane pods go into a crash loop after changing the --cluster-cidr in the yaml.

The output in the controller-manager logs are below?

controllermanager.go:235] error starting controllers: failed to mark cidr[192.168.0.0/24] at idx [0] as occupied for node: : cidr 192.168.0.0/24 is out the range of cluster cidr 10.0.0.0/16

Labret

Answer 1

Changing a cluster CIDR isn't a simple task. I managed to reproduce your scenario and I managed to change it using the following steps.

Changing an IP pool

The process is as follows :

1. Install calicoctl as a Kubernetes pod (Source)
2. Add a new IP pool (Source).
3. Disable the old IP pool. This prevents new IPAM allocations from the old IP pool without affecting the networking of existing workloads.
4. Change nodes podCIDR parameter (Source)
5. Change --cluster-cidr on kube-controller-manager.yaml on master node. (Credits to OP on that)
6. Recreate all existing workloads that were assigned an address from the old IP pool.
7. Remove the old IP pool.

Let’s get started.

In this example, we are going to replace 192.168.0.0/16 to 10.0.0.0/8.

1. Installing calicoctl as a Kubernetes pod

   $ kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
   

   Setting an alias:

   $ alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl "
   

2. Add a new IP pool:

   calicoctl create -f -<<EOF
   apiVersion: projectcalico.org/v3
   kind: IPPool
   metadata:
     name: new-pool
   spec:
     cidr: 10.0.0.0/8
     ipipMode: Always
     natOutgoing: true
   EOF
   

   We should now have two enabled IP pools, which we can see when running calicoctl get ippool -o wide:

   NAME                  CIDR             NAT    IPIPMODE   DISABLED
   default-ipv4-ippool   192.168.0.0/16   true   Always     false
   new-pool              10.0.0.0/8       true   Always     false
   

3. Disable the old IP pool.

   First save the IP pool definition to disk:

   calicoctl get ippool -o yaml > pool.yaml
   

   pool.yaml should look like this:

   apiVersion: projectcalico.org/v3
   items:
   - apiVersion: projectcalico.org/v3
     kind: IPPool
     metadata:
       name: default-ipv4-ippool
     spec:
       cidr: 192.168.0.0/16
       ipipMode: Always
       natOutgoing: true
   - apiVersion: projectcalico.org/v3
     kind: IPPool
     metadata:
       name: new-pool
     spec:
       cidr: 10.0.0.0/8
       ipipMode: Always
       natOutgoing: true
   

   Note: Some extra cluster-specific information has been redacted to improve readibility.

   Edit the file, adding disabled: true to the default-ipv4-ippool IP pool:

   apiVersion: projectcalico.org/v3
   kind: IPPool
   metadata:5
     name: default-ipv4-ippool
   spec:
     cidr: 192.168.0.0/16
     ipipMode: Always
     natOutgoing: true
     disabled: true
   

   Apply the changes:

   calicoctl apply -f pool.yaml
   

   We should see the change reflected in the output of calicoctl get ippool -o wide:

   NAME                  CIDR             NAT    IPIPMODE   DISABLED
   default-ipv4-ippool   192.168.0.0/16   true   Always     true
   new-pool              10.0.0.0/8       true   Always     false
   

4. Change nodes podCIDR parameter:

   Override podCIDR parameter on the particular k8s Node resource with a new IP source range, desirable way with the following commands:

   $ kubectl get no kubeadm-0 -o yaml > file.yaml; sed -i "s~192.168.0.0/24~10.0.0.0/16~" file.yaml; kubectl delete no kubeadm-0 && kubectl create -f file.yaml
   $ kubectl get no kubeadm-1 -o yaml > file.yaml; sed -i "s~192.168.1.0/24~10.1.0.0/16~" file.yaml; kubectl delete no kubeadm-1 && kubectl create -f file.yaml
   $ kubectl get no kubeadm-2 -o yaml > file.yaml; sed -i "s~192.168.2.0/24~10.2.0.0/16~" file.yaml; kubectl delete no kubeadm-2 && kubectl create -f file.yaml    
   

   We had to perform this action for every node we have. Pay attention to the IP Ranges, they are different from one node to the other.

5. Change CIDR on kubeadm-config ConfigMap and kube-controller-manager.yaml

Edit kubeadm-config ConfigMap and change podSubnet to the new IP Range:

kubectl -n kube-system edit cm kubeadm-config

Also, change the --cluster-cidr on /etc/kubernetes/manifests/kube-controller-manager.yaml located in the master node.

$ sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.0.0.0/8
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true

7. Recreate all existing workloads using IPs from the disabled pool. In this example, kube-dns is the only workload networked by Calico:

   kubectl delete pod -n kube-system kube-dns-6f4fd4bdf-8q7zp
   

   Check that the new workload now has an address in the new IP pool by running calicoctl get wep --all-namespaces:

   NAMESPACE     WORKLOAD                   NODE      NETWORKS            INTERFACE
   kube-system   kube-dns-6f4fd4bdf-8q7zp   vagrant   10.0.24.8/32   cali800a63073ed
   

8. Delete the old IP pool:

   calicoctl delete pool default-ipv4-ippool
   

Creating it correctly from scratch

To deploy a cluster under a specific IP range using Kubeadm and Calico you need to init the cluster with --pod-network-cidr=192.168.0.0/24 (where 192.168.0.0/24 is your desired range) and than you need to tune the Calico manifest before applying it in your fresh cluster.

To tune Calico before applying, you have to download it's yaml file and change the network range.

1. Download the Calico networking manifest for the Kubernetes.

   $ curl https://docs.projectcalico.org/manifests/calico.yaml -O
   

2. If you are using pod CIDR 192.168.0.0/24, skip to the next step. If you are using a different pod CIDR, use the following commands to set an environment variable called POD_CIDR containing your pod CIDR and replace 192.168.0.0/24 in the manifest with your pod CIDR.

   $ POD_CIDR="<your-pod-cidr>" \
   sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml
   

3. Apply the manifest using the following command.

   $ kubectl apply -f calico.yaml
   

Hesitancy

If we initially made the cluster following the steps above, and later realize that we need to extend the range to include more IP addresses, is there a way to make this change? We have deployed this way and have the mentioned need, without needing to rebuild the entire cluster. – Labret

I updated my answer including steps on how to change it without needing to rebuild the cluster. – Hesitancy

when hard coding the pod CIDR in the nodes in step 4 (this is done on master and worker nodes, correct?) whenever we go to add additional worker nodes to the cluster, will they automatically assign the next ip in the range? for example, in the above master 1 is 192.168.0.0, master 2 is 192.168.1.0, master3 is 192.168.2.0 ..... the worker nodes continue in this pattern for 3, 4, 5, etc. will the next worker node pick up this range and be assigned 6 (or the next available ip in the range? Also, does this need to be done in a particular order, to make sure that we maintain masters up at all times – Labret

So... in addition to the above, it looks like you need to manually edit the /etc/kubernetes/manifests/kube-controller-manager.yml to the new cidr. once this is changed, it looks like the change is complete – Labret

I will validade It and if really necessary, I will update my answer including.it with credits.to you. Glad you managed to complete the change – Hesitancy

Another question about the ranges. Is it okay if the ranges for the nodes overlap? Or should the nodes have a /24 as each node would be assigned a range within the overall calico pod cidr range? – Labret

regarding the above comment, I am thinking about the 10.0.0.0/16, 10.0.1.0/16, and 10.0.2.0/16. Since the /16 range would now include each other since it is not just a /24 per node – Labret

It makes sense for me. Probably would.be a very good idea to use 10.0.0.0/16, 10.1.0.0/16 and 10.2.0.0/16. – Hesitancy

I edited my answer including the points you mentioned and an additional change in the kubeadm-config ConfigMap. – Hesitancy

I am confused by the IP ranges bring used in the example. The IP pool defined within calico is 10.0.0.0/16, but the nodes are using 10.1.0.0/16 and 10.2.0.0/16 - these IPs are not included in the range of the calico IP pool, so how are they being used for the node variable for the IP range? Can you confirm that the above configuration is correct? – Labret

I believe the link below is the reason that this works. the calico IPAM does not follow the pod cidr specified on the node, rather it uses its own range and pushes it out to the nodes: github.com/projectcalico/calico/issues/2592 – Labret

@mm_wvu18 thanks for raising this topic. I reviewed it and realized that you are right, they can't be in the same network as i described. I edited the guide basing it on a change from 192.168.0.0/16 to 10.0.0.0/8. This makes more sense and will make this guide a lot easier to understand. – Hesitancy

I am not sure that this variable matters in the first place, as it seems that the CIDRs on the nodes are assigned and picked up randomly (not based off of the IP that you assign within the "file.yaml" used above. This may be using the description in my previous comment (regarding Calico IPAM). – Labret

Answer 2

In addition to accepted answer, In my case, I have to change ipv4_pools parameter of calico-ipam in this file: /etc/cni/net.d/10-calico.conflist and /etc/cni/net.d/calico.conflist.template

In summary, in all nodes:
vim /etc/cni/net.d/10-calico.conflist
vim /etc/cni/net.d/calico.conflist.template

{
  "name": "k8s-pod-network",
  "cniVersion":"0.3.1",
  "plugins":[
    {
      "datastore_type": "kubernetes",
      "nodename": "node-1",
      "type": "calico",
      "log_level": "info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "ipam": {
        "type": "calico-ipam",
        "ipv4_pools": ["192.168.0.0/24"], #CHANGE HERE!! e.g. 10.0.0.0/8
        "assign_ipv4": "true"
      },
      "policy": {
        "type": "k8s"
      },
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
...

Mccormac