Is there any way to bind K3s / flannel to another interface?

Question

I have a K3s (v1.20.4+k3s1) cluster with 3 nodes, each with two interfaces. The default interface has a public IP, the second one a 10.190.1.0 address. I installed K3s with and without the -flannel-backend=none option and then deployed flannel via " kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml", previously binding the kube-flannel container to the internal interface via the args "--iface=". In this setup the kube-flannel pods get the node-ip of the internal interface, but I can't reach the pods on the other nodes via ICPM. If I deploy flannel without -iface arg, the kube-flannel pods get an address from the 10.42.0.0 network. Then I can reach the pods of the other hosts, but the traffic will be routed through the public interfaces, which I want to avoid. Does anyone have a tip for me?

Bezoar

Is ICMP traffic enabled in your private network ? Did you try to reach these Pods from a node or from another Pod ? Have you tried to use --node-ip argument ? As far as I know, there is --flannel-iface argument, not --iface=. – Powerless

you are right, "--flannel-iface=" worked. thank you! – Bezoar

That's great :) so using --flannel-iface= solved your problem and now everything works as expected ? – Powerless

exactly :) now i can use: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.20.4+k3s1" INSTALL_K3S_EXEC="server --disable=traefik --flannel-iface=ens10 --bind-address 10.190.1.5 --advertise-address 10.190.1.5 --datastore-endpoint etcd --node-name kube-master-01 --node-ip 10.190.1.5 --node-external-ip 10.190.1.30 --cluster-init" sh -s - to install the first master node, while ens10 ist the second interface – Bezoar

Answer 1

The problem was resolved in the comments section but for better visibility I decided to provide an answer.

As we can see in the K3s documentation, K3s uses flannel as the CNI by default:

By default, K3s will run with flannel as the CNI, using VXLAN as the default backend. To change the CNI, refer to the section on configuring a custom CNI.

By default, flannel selects the first interface on a host (look at the flannel documentation), but we can override this behavior with the --flannel-iface flag.
Additionally we can explicitly set IP address to advertise for node using the --node-ip flag.

---

I've created a simple example to illustrate how it works.

On my host machine I have two network interfaces (ens4 and ens5):

kmaster:~# ip a s | grep -i "UP\|inet"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.156.15.197/32 brd 10.156.15.197 scope global dynamic ens4
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.0.2/32 brd 192.168.0.2 scope global dynamic ens5

Without setting the --flannel-iface and --node-ip flags, flannel will select the first interface (ens4: 10.156.15.197):

kmaster:~# curl -sfL https://get.k3s.io |  sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.20.4+k3s1 as release
...
[INFO]  systemd: Starting k3s
kmaster:~# kubectl get nodes -o wide  
NAME      STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     
kmaster   Ready    control-plane,master   97s   v1.20.4+k3s1   10.156.15.197

But as I mentioned before we are able to override default flannel interface with the --flannel-iface flag:

kmaster:~# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-ip=192.168.0.2 --flannel-iface=ens5" sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.20.4+k3s1 as release
...
[INFO]  systemd: Starting k3s
kmaster:~# kubectl get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   
kmaster   Ready    control-plane,master   64s   v1.20.4+k3s1   192.168.0.2 

Powerless

can a specific pod or service connect through ens5 for example ( to make a request for a server in the local network) and reach the other pods using the ens4 interface ? – Corduroy

Answer 2

Thanks to @matt_j for the solution. It worked for me. you can apply these changes after installation of k3s cluster with just changing the systemd k3s.service on master nodes and k3s-agent.service on worker nodes.

Problem: Installing k3s server without extra options will cause flannel to choose default network interface that in my case it was 'enp0s3' and I wanted 'enp0s8' be the default network interface for flannel.

Note: In this example master_hostname=docker01 , master_ip=192.168.56.10 and worker_hostname=docker02 , worker_ip=192.168.56.20

sudo nano /etc/systemd/system/k3s.service

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
        '--write-kubeconfig' \
        '/home/docker01/.kube/config' \
        '--write-kubeconfig-mode' \
        '666' \
        '--tls-san' \
        '192.168.56.10,192.168.56.20,docker02' \
        '--node-external-ip=192.168.56.10' \
        '--flannel-external-ip' \
        '--node-ip' \
        '192.168.56.10' \
        '--flannel-iface' \
        'enp0s8' \

sudo nano /etc/systemd/system/k3s-agent.service

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    agent \
        '--token' \
        'K10f0943a0652c34c251102669ea4fde4e777a1971c69c1ed8c07e7ee05ab630e2f::server:4298a71209a00719551ee49d868c6fba' \
        '--server' \
        'https://192.168.56.10:6443' \
        '--node-external-ip' \
        '192.168.56.20' \
        '--flannel-iface' \
        'enp0s8' \

After changing your ExecStart , you can apply changes on all nodes by:

sudo systemctl daemon-reload

sudo systemctl restart <k3s on master node or k3s-agent on worker node>

Borrell

Very nice. This can also be achieved via configuration files . – Puzzle