docker change cgroup driver to systemd

Question

I want Docker to start with systemd cgroup driver. For some reason it is using only cgroupfs on my CentOS 7 server.

Here is startup config file.

# systemctl cat docker
# /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer

[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
          --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
          --default-runtime=docker-runc \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/docker.service.d/docker-thinpool.conf
 [Service]
 ExecStart=
 ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool \
 --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
 EOF

When I start Docker, it's running like this:

# ps -fed | grep docker
root      8436     1  0 19:13 ?        00:00:00 /usr/bin/dockerd-current --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true
root      8439  8436  0 19:13 ?        00:00:00 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc

Here is the output of docker info:

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 185.6 MB
 Data Space Total: 1.015 GB
 Data Space Available: 829.4 MB
 Metadata Space Used: 77.82 kB
 Metadata Space Total: 8.389 MB
 Metadata Space Available: 8.311 MB
 Thin Pool Minimum Free Space: 101.2 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.16.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 992.7 MiB
Name: master
ID: 6CFR:H7SN:MEU7:PNJH:UMSO:6MNE:43Q5:SF4K:Z25I:BKHP:53U4:63SO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Registries: docker.io (secure)

How can I make it run with systemd?

Thanks SR

Jedediah

Answer 1

Since I have two configuration file I need to add the entry in the second config file also -- /etc/systemd/system/docker.service.d/docker-thinpool.conf:

--exec-opt native.cgroupdriver=systemd \

Jedediah

Thank you :) I've been looking for this for a while. In my case, I only added the --exec-opt native.cgroupdriver=systemd an additional option directly to the docker.service (after copying it to /etc/systemd/system/, in order to keep it persistent after docker ugprades). – Colt

failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd". - getting this error, so changing docker Cgroup Driver: cgroupfs – Papua

Answer 2

A solution that does not involve editing systemd units or drop-ins would be to create (or edit) the /etc/docker/daemon.json configuration file and to include the following:

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

After saving it, restart your docker service.

sudo systemctl restart docker

This solution obviously is only feasible if you would want to apply this system-wide.

Bluefield

If you're appending to existing contents in the file /etc/docker/daemon.json, make sure you put a comma to the last line before adding the above line. – Botelho

Answer 3

Just to add, cgroupfs is dockers own control group manager. However, for the majority of Linux distributions ssytemd is the default init system now and systemd has tight integration with Linux control groups and In Kubernetes site, they recommend using systemd (see below) as using cgroupfs along with systemd seems to be non-optimal

So it is better to use systemd then for cgroup managment. kubelet is configured by default to use systemd. So it is easier and better to change Docker to use the systemd Cgroup driver

A history of this overlap is here https://lwn.net/Articles/676831/

In Kubernetes site, they recommend using systemd https://kubernetes.io/docs/setup/production-environment/container-runtimes/

Cgroup drivers When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager. Systemd has a tight integration with cgroups and will allocate cgroups per process. It’s possible to configure your container runtime and the kubelet to use cgroupfs. Using cgroupfs alongside systemd means that there will then be two different cgroup managers.

Control groups are used to constrain resources that are allocated to processes. A single cgroup manager will simplify the view of what resources are being allocated and will by default have a more consistent view of the available and in-use resources. When we have two managers we end up with two views of those resources. We have seen cases in the field where nodes that are configured to use cgroupfs for the kubelet and Docker, and systemd for the rest of the processes running on the node becomes unstable under resource pressure.

Pontiac

why when we speak about kubernetes we set cgroups driver to systemd onlly for kubelet if kubernetes includes many components? – Barragan

Answer 4

OS: Centos 7.4 As kubernetes 1.23.1 recommend to use cgroup systemd, and docker 20.10.20 use cgroup cgroupfs. So, you have to change docker service file.

step1: Stop docker service

systemctl stop docker

step2: change on files /etc/systemd/system/multi-user.target.wants/docker.service and /usr/lib/systemd/system/docker.service

From :

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

TO:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

step3: start docker service and kubelet

systemctl start docker

kubeadm init phase kubelet-start

Levo

there should be a systemctl daemon-reaload command before step3 – Paper

It is bad practice to overwrite systemd configuration files supplied by vendors. These files will be overwritten by a package update at some point in the future. Instead you should override settings via drop-ins. Better yet, look at my previous answer and change the setting via Docker’s daemon.json system-wide configuration file. – Bluefield

Answer 5

Make sure you are logged in as root and execute the below two commands :

1. echo '{"exec-opts": ["native.cgroupdriver=systemd"]}' >> /etc/docker/daemon.json
2. systemctl restart docker

Constituency

The approach is correct, but realization is not. Echo message does not take in account existing JSON configuration in daemon.json – Maurene

Answer 6

Try to restart the docker service:

systemctl daemon-reload
systemctl restart docker.service

Affectional

and automagically it will select another cgroup driver? – Verminous