Docker change mtu on the fly?

Question

I have production docker swarm with 9 stacks, most of them have volumes. Currently docker is running with single node.

I have to add second node and it is the place where problems started. Especially the problem is with portainer - it becomes very laggy, almost unusable. Also when I move some containers to a new node, my project seems to be completely stucked - the problem is in communication between containers on different nodes. Some requests are ok, but most of them seems to be broken.

After some research I found out that problem seems to be with MTU. MTU of eth1 is 1450 and the docker's default is 1500.

The question is, is it possible to change docker's MTU to 1450 on the fly?

I tried:

1. Add key to dockerd --mtu=1450 - the docker service didn't start at all
2. Change main network's mtu in docker-compose - seems that it didn't updated, I think that the network should be recreated
3. Add mtu option to /etc/docker/daemon.json also seemed to be with no effect

How to change mtu on working production server? Possible downtime for 10-15 minutes is ok, but I don't want to remove all stacks and recreate them.

Cape

Answer 1

Solved. The long way...

I had vps named "m1" with my docker stacks and finally yesterday I realized that I can not update mtu on working cluster ((

So I added "m2" (as manager) and "m3" (as worker) and created docker swarm cluster on "m2"+"m3" (not "m1").

1. I modified /lib/systemd/system/docker.service on "m2" and added --mtu=1450 here:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --mtu=1450

I also created new file /etc/docker/daemon.json with { "mtu": 1450 } (thanks @BMitch)

2. I removed ingress network on "m2" and recreated it with option "com.docker.network.driver.mtu": "1450"

3. I added mtu to all my overlay networks in project (in docker-compose):

networks:
  network1:
    driver: overlay
    driver_opts:
      com.docker.network.driver.mtu: 1450

Update: here is the purpose of "m2"+"m3" - I was able check that the problem is solved at new cluster, while my production "m1" was working. I tried portainer and it was working without any lags I've seen before. Next time I will just remove stacks on laggy vps, update settings (mtu) and redeploy stacks - this would be much more faster!

4. I removed stacks from "m1" and copied volumes to "m2" (thanks to How to copy docker volume from one machine to another?)

5. I deployed services to "m2" and updated my domain's DNS to "m2" ip

It works fine with no lags after MTU update!

Cape

"I modified /lib/systemd/system/docker.service" ... that will be overwritten on the next upgrade. Edit /etc/docker/daemon.json with the contents { "mtu": 1450 } (or add to the existing contents of the file). – Caterina

Thanks for this notice, I will update my post – Cape

An even better solution would be to use systemctl edit docker, which will create an override unit alongside the original service. That will survive updates. – Endamage

Thanks for the great answers folks! I wonder why docker (the docker0 network) doesn't default to the host server's (eth0) MTU 9000. And seems weird I have to set in both daemon.json and in docker-compose.yml, why doesn't the network created by compose (v2.19) use the system default? – Lungki

Maybe there are many interfaces, so docker cannot choose one? ) – Cape