docker.socket: Failed with result 'service-start-limit-hit' after protecting docker daemon socket

Question

I followed the steps provided in the documentation here to add tls security for docker api. Certificates are located in ~/.docker/ as well as /etc/docker/ssl/ folders. I added override.conf to /etc/systemd/system/docker.service.d/ with content

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem

Then, I used daemon-reload and docker start

$ systemctl daemon-reload
$ service docker start

The errors in journalctl -xe is:

-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Start request repeated too quickly.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 15 21:43:24 cynicalplyaground systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit docker.service has failed.
-- 
-- The result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 15 21:45:01 cynicalplyaground CRON[12769]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session closed for user root

How can I sort this issue?

Counteroffensive

Answer 1

In the present case the same error occured after the latest manjaro update (2020-01-20).
Tried to change the systemd docker service, as adviced in other cases, but I reverted those changes and finally this was solved with:

• a reboot of the system

(like advised here: https://www.reddit.com/r/archlinux/comments/7ya4ug/installing_docker_on_arch_linux/)

Humanly

A reboot worked for me.. – Stacey

A reboot worked for me too – Extravagancy

Had the same issue with PI, worked !! – Haem

Answer 2

Getting to the root of the problem;

systemctl status docker.service

has this: /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Trying to run that command, it complains about unable to configure the Docker daemon with file /etc/docker/daemon.json: EOF

ls -l /etc/docker/daemon.json -rw-r--r-- 1 root root 0 Jul 30 10:32 /etc/docker/daemon.json

NOTE that the JSON file is empty. Delete it.

Tobit

I got this issue when disabling a plugin and still having an existing daemon.json file, so after removing it and restarting docker it works again. Thanks. – Apace

I had a similar issue, but in my case it was /run/containerd/containerd.sock. Restarting containerd fixed it. – Thermolysis

Same here. Running dockerd --debug revealed a formatting issue with /etc/docker/daemon.json. – Lohner

Somehow daemon.json contained an unexpected "graph" key, perhaps after an update. Removing it solved the issue. – Filar

I was getting docker.service: Start request repeated too quickly. but decided to backup the file and try this anyway. It worked ¯\_(ツ)_/¯ – Conformable

This helped me! I couldn't reboot the server because other services run on it and I'm not the server manager. Fixing my daemon.json fixed the issue. – Hanahanae

Answer 3

For me it was because the docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

dockerd, should start fine after switching to iptables-legacy.

Phantasm

Answer 4

I have the same issue and just modify the "/usr/bin/dockerd" to "/usr/sbin/dockerd", then it works. You can check the dockerd path first.

Calculating

Answer 5

I faced a similar issue on Ubuntu because I added the hosts option to /etc/docker/daemon.json file. That's ok, but for systems that use systemd it may cause conflict with the arguments passed to dockerd on start.

The solution was to delete the /etc/docker/daemon.json's hosts entry and set this config on file /etc/systemd/system/docker.service.d/options.conf.

$ cat /etc/systemd/system/docker.service.d/options.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://

After that, restart the service.

$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

You may check that your changes has been applied by running docker info. Also, you may note on the docker service status that Drop-In field is using the options.conf created, and dockerd was executed with the specified host list.

$ systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset>
    Drop-In: /etc/systemd/system/docker.service.d
             └─options.conf
     Active: active (running) since Fri 2022-11-18 01:02:18 EST; 1h 50min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1111 (dockerd)
      Tasks: 18
     Memory: 58.5M
        CPU: 1.294s
     CGroup: /system.slice/docker.service
             └─1111 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://

References:

• Daemon configuration file
• Control Docker with systemd

Burnard

Answer 6

I had a similar issue on nixOS installed in a btrfs filesystem.

For me the solution was to add virtualisation.docker.storageDriver = "btrfs"; to my /etc/nixos/configuration.nix

Which according to the docker docs should equate to adding the following to /etc/docker/daemon.json in most other distros:

{
  "storage-driver": "btrfs"
}

Originally

Answer 7

I had same problem on CentOS 7 after upgrade of obsolete docker package to docker-ce. It appeared that bridge docker0 left enabled in firewall conf after old package removal and docker service start failed when attempted to enable already enabled interface.

sudo firewall-cmd --zone trusted --remove-interface docker0

solved the problem.

Flout

Answer 8

in my case... the host was part of a docker swarm...but the IPv6 was no longer reachable or automatically assigned to the host... I manually add the old_IPv6

ip -6 address add 28xx:xxxx:x:x:xx:ebff:fe14:xxx dev ens3x

the journalctl -u docker.service mention:

level=fatal msg="Error starting cluster component: could not find local IP address: dial udp [2xxx:xxx:xxxx:xxx]:2377: connect: network is unreachable"

after add manually the IPv6 I was able to start docker so with docker running I leave the "swarm" and reboot

docker swarm leave --force

after reboot the docker services run as usual

Sea

Answer 9

For me it was missing disk space. Reboot also helped, but I was stillnot able to build any container.

After pruning some outdated stuff from the docker volumes I was able to continue.

Rubidium

Answer 10

I had a similar issue and tried rebooting as above and also modifying the docker.service to remove the -H fd:// argument as well.

However, I kept getting the following:

sudo systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset:>
     Active: failed (Result: service-start-limit-hit) since Sun 2023-03-26 20:3>
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)

Mar 26 20:38:58 kourt systemd[1]: Starting Docker Socket for the API.
Mar 26 20:38:58 kourt systemd[1]: Listening on Docker Socket for the API.
Mar 26 20:39:07 kourt systemd[1]: docker.socket: Failed with result 'service-st>

Closer inspection shows that the socket is listening at /run/docker.sock but prompted me to look in /var/run. There (due to a previous version) I had a folder called /var/run/docker/plugins.

Removing /var/run/docker enabled the docker.socket service to start and and enable docker.service to start.

Running Ubuntu 20.04.6 LTS.

Clack

Answer 11

I got this error when I tried to move the root dir of docker, and it lead me to this thread. If you followed a guide to move the base directory of docker, which is normally located at /var/lib/docker to somewhere else, like <new_path>/docker, you might have also run into this problem. The reason is that some guides, like the one I linked, use an older version of docker, which uses the -g shorthand to specify the new path, but in more recent versions that flag doesn't exist, and you have to use --data-root instead. So if you changed the data root in the file /lib/systemd/system/docker.service to:

ExecStart=/usr/bin/dockerd -g <new_path>/docker -H fd:// --containerd=/run/containerd/containerd.sock

It should actually be:

ExecStart=/usr/bin/dockerd --data-root <new_path>/docker -H fd:// --containerd=/run/containerd/containerd.sock

Wendall

Answer 12

I was able to solve the problem by disabling the firewalld

systemctl disable firewalld
systemctl stop firewalld

Casimir