cannot start docker daemon (graphdriver issue)

Question

running Ubuntu 16.04.5 LTS 4.4.0-108-generic on a virtual private server My aim is to deploy a meteor application with mup.js, but it fails because of dockerd not being launched. The problem is, i cannot get dockerd to launch after a system reboot.

I tried changing graphdriver as suggested in other threads (Not able to start docker on Ubuntu 16.04.2 LTS (error initializing graphdriver)), switching to aufs or overlay2, but to no avail. I also updated my kernel, purged docker repos, reinstalled docker on my machine.

i have close to no experience working with docker, and the website i'm trying to put back online is part of a show, the last night of which is tomorrow! i must say i'm getting a bit desperate, any help is welcome.

thank you!

docker & dockerd are both version 18.06.1-ce, build e68fc7a

$ sudo dockerd

INFO[0000] libcontainerd: new containerd process, pid: 3488 
WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=1048576
WARN[0000] failed to rename /var/lib/docker/tmp for background deletion: %!s(<nil>). Deleting synchronously 
Error starting daemon: error initializing graphdriver: driver not supported

journalctl -xe yields :

Oct 03 01:22:19 vps332343 systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is done.
Oct 03 01:22:19 vps332343 systemd[1]: docker.service: Start request repeated too quickly.
Oct 03 01:22:19 vps332343 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Oct 03 01:22:19 vps332343 systemd[1]: docker.socket: Unit entered failed state.
Oct 03 01:22:37 vps332343 sudo[3651]:  eboutin : TTY=pts/0 ; PWD=/etc/nginx/sites-available ; USER=root ; COMMAND=/bin/journalctl -xe
Oct 03 01:22:37 vps332343 sudo[3651]: pam_unix(sudo:session): session opened for user root by eboutin(uid=0)

df -tH yields :

Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  970M     0  970M   0% /dev
tmpfs          tmpfs     196M  5.6M  190M   3% /run
/dev/vda1      ext4      9.7G  4.6G  5.1G  48% /
copymods       tmpfs     977M   28K  977M   1% /lib/modules
tmpfs          tmpfs     977M   68K  977M   1% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     977M     0  977M   0% /sys/fs/cgroup
tmpfs          tmpfs     196M     0  196M   0% /run/user/1002
tmpfs          tmpfs     196M     0  196M   0% /run/user/1001

/etc/docker/daemon.json contents :

{"storage-driver":"devicemapper"}

(no other modified config file)

Angola

What is your kernel version? How have you installed docker and what is your dockerd version? – Lasseter

Also, please include you /etc/docker/daemon.json along with any other modified config files. – Lasseter

well hum hehe we are currently running on kernel 4.4.0-108-generic after downgrading from 4.4.0-137 (we edited the grub file and purged the more recent kernels..). I edited my post with the rest of the info. – Angola

What happens if you remove your daemon.json, or switch to overlay2? – Lasseter

deleting the json file and trying to launch deamond throws the same graphdriver error. (ERRO[2018-10-03T16:11:41.088732386+02:00] [graphdriver] prior storage driver devicemapper failed: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed Error starting daemon: error initializing graphdriver: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed) – Angola

making a new json file with {"storage-driver":"overlay2"} also throws a graphdriver error : ERRO[2018-10-03T16:08:04.961130302+02:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay2 – Angola

Strange, devicemapper shouldn't be the default on Ubuntu. Could be seeing some files in /var/lib/docker that is forcing it to go back to that. Also looking like something isn't right with the kernel, perhaps a version that has the needed modules disabled. – Lasseter

Answer 1

Try to configure devicemapper as a storage driver and clean /var/lib/docker/ folder before docker start rm -rf /var/lib/docker/* (it will delete all your previous containers/volumes/...).

Check any warnings from docker info, when docker will be running - they may help you with additional configuration.

Encyclopedia

i now have the following error : Error starting daemon: error initializing graphdriver: devicemapper: Error running deviceCreate (CreatePool) dm_task_run failed – Angola

Do you have any free space? – Encyclopedia

Aufs is default option in Ubuntu, and it works well. Wondering why you suggesting so? – Pliant

thank you for your answers. I have root, hum i have to say that i don't know what is my fs (file system?). I have free space but not a massive amount (maybe 5G) as it's a small droplet. regarding aufs, i can't manage to get it installed either, nor does dockerd when i specify it as a graphdriver.. – Angola

devicemapper is the most troublefree storage driver. But this looks like an OS issue - maybe you have a custom kernel without full docker support, maybe /var/lib/docker is not mounted with correct parameters, .... 'df -Th' command can be used to check fs type/size. – Encyclopedia

i reinstalled docker already, so var/lib/docker should be clean? i'm going to look into maybe upgrading the kernel, also i edited post to show the results of df -Th. – Angola

Devicemapper is supported for rhel and centos, and is being phased out because of all the issues and limitations it has. This shouldn't be used on a Ubuntu install. – Lasseter

This seems to an issue not related to used storage driver. Anyway, yes devicemapper has own issues, but other storage drivers have as well. It is officially supported on Ubuntu as well - docs.docker.com/storage/storagedriver/select-storage-driver/… – Encyclopedia

It can be supported and phased out at the same time. Everything is defaulting to overlay2 with a new enough kernel now. – Lasseter

Answer 2

This can also be due to a recent kernel update that might have messed up the graphdriver: devicemapper.

So when rm -rf /var/lib/docker/* and reinstalling Docker does not work. Try reinstalling kernel image and reboot.

$ sudo apt-get install --reinstall linux-image-`uname -r`
$ sudo reboot

Edifice

Answer 3

I have just removed the folder devipemapper under that root directory and reload the daemon and restarted. it worked for me.

Curet

This also worked for me on PopOS 20.04, multiple re-installs + kernels + updates and it was this folder sticking around. – Applaud

Answer 4

We are using Ubuntu 20.04 LTS. At one point, we needed to change the config to use the devicemapper settings. Now, devicemapper is deprecated. We simply removed the config file (the only settings was for the storage settings) and the default overlay2 appears to work just fine.

Uprising