Can't run Docker container due device mapper error

Question

I just can't create and run new containers in Docker anymore. But in the same time a can run previously created containers.

When I try to do something like this:

[user@host ~ ] docker run --name=fpm-5.3 debian:jessie
2014/07/12 07:34:08 Error: Error running DeviceCreate (createSnapDevice) dm_task_run failed

From docker.log:

2014/07/12 05:57:11 POST /v1.12/containers/create?name=fpm-5.3
[f56fcb6f] +job create(fpm-5.3)
Error running DeviceCreate (createSnapDevice) dm_task_run failed
[f56fcb6f] -job create(fpm-5.3) = ERR (1)
[error] server.go:1025 Error: Error running DeviceCreate (createSnapDevice) dm_task_run failed
[error] server.go:90 HTTP Error: statusCode=500 Error running DeviceCreate (createSnapDevice) dm_task_run failed

dmsetup status

docker-8:1-1210426-pool: 0 209715200 thin-pool 352 2510/524288 205173/1638400 - ro discard_passdown queue_if_no_space 

But they are a lot of free space on disk.

dmsetup info

Name:              docker-8:1-1210426-pool
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      1
Major, minor:      252, 0
Number of targets: 1

docker info

Containers: 4
Images: 65
Storage Driver: devicemapper
 Pool Name: docker-8:1-1210426-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 12823.3 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 9.9 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.14.4

docker version

Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.2
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.2
Git commit (server): 63fe64c

Redbug

Answer 1

The following is for a Fedora/RHEL system, so you'll need to adjust for Debian...

# systemctl stop docker.service
# thin_check /var/lib/docker/devicemapper/devicemapper/metadata

If there were no errors then proceed with:

# thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata
# systemctl start docker.service
# docker run --name=fpm-5.3 debian:jessie

---

To install thin_check command, run:

# apt-get install -y thin-provisioning-tools

Titania

Thanks. During this time I've been switched to aufs. But after run thing_check everything is ok. And I think, it would be useful to make some 'storage info page' about switching storage backends and problems about this on Docker Docs. – Redbug

what should I do if thin_check returns superblock is corrupted? Is there still a way to fix it? Thanks – Maurya

Answer 2

When the docker partition filled and docker would no longer start after reboot, I encountered this:

# thin_check /var/lib/docker/devicemapper/devicemapper/metadata
examining superblock
examining devices tree
  missing devices: [0, -]
    bad checksum in btree node
examining mapping tree
  thin device 72 is missing mappings [137494, 137594]
    bad checksum in btree node
  thin device 72 is missing mappings [137721, -]
    bad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree nodebad checksum in btree node

I was able to repair with this procedure:

# thin_dump -r /var/lib/docker/devicemapper/devicemapper/metadata -o /tmp/metadata.xml
# thin_restore -i /tmp/metadata.xml -o /var/lib/docker/devicemapper/devicemapper/metadata

Groundage

dump and then restore, this is working. – Dietsche

Answer 3

In our case just need to start and stop the service.

Depending on your system you can run:

service docker stop
service docker start

or:

sudo systemctl stop docker.service
sudo systemctl start docker.service

or:

sudo /etc/init.d/docker restart

Supernatural

Answer 4

I had the same problem and was not able to fix it. I found something promising at: http://grokbase.com/t/gg/docker-user/1563fzdtm7/docker-docker-runs-out-of-space-when-trying-to-create-a-new-image 'The default docker storage driver allocates 10GB storage blocks for your images. Move to overlayfs and avoid this entirely. In the command that starts your docker daemon just add "-s overlay" '

this fixed my problem.

Systematist

Unfortunately you cannot use rpm or yum with OverlayFS bugzilla.redhat.com/show_bug.cgi?id=1213602 – Hark

Answer 5

I've been fighting this issue with Debian 8.2. I had other problems because I run a 4.3.3 Kernel (default is 3.16) with grsec.

Despite the GRSEC issues (mount & chmod denied) I was able to run docker and create some image and container.

Then, I would reboot and docker would just spit out the error. I ran thin_check and what I found was this:

• metadata was fine (I don't use a thin pool right now since docker keeps on telling me that my thin pool isn't a thin pool...)
• data was corrupted.

Tried to repair it but thin_restore crashes.

I realized that: docker daemon ... was working but can't be stopped with systemctl stop docker.service. It says that the service is stopped but the daemon is still in memory (ps -elf | grep docker)

To fix the problem I had to change the DOCKER_STORAGE_OPTIONS in /etc/default/docker

rm -rf /var/lib/docker
reboot

At boot time, the service starts. docker info

Shows the info as expected. Built an image. Rebooted, the service starts fine again. I think that basically the docker daemon can't be stopped and killing it with a:

kill <pid>

Causes the data file to be corrupted and therefore the checksum won't match.

Bottom line is don't mix and match docker.service and docker daemon. At least on Debian/Ubuntu.

Armendariz

I can confirm that docker can be "unstable" in some devops testing cases on Jessie, but it's a systemd & Debian problem right now. But on latest Docker and aufs (4.4.0-r1 kernel) everything works like a charm (mainly on amazon linux, Debian and Gentoo). So I'll suggest you switch to aufs, it's much more stable for me than dm, anyway you just can take a peek. – Redbug

Unfortunately I can't switch to aufs since I'm required to run GRSEC and the aufs patch doesn't like the GRSEC patch. I successfully switched to a thin pool but that's even worse as the pool gets corrupted and udev can't mount it anymore. Even if I stop the containers before rebooting, the pool still gets screwed. – Armendariz

So, seems that GRSEC breaks some stuff here. Maybe just put machine where Docker runs in DMZ or something like this? In my experience Docker very sensitive to kernel build flags and especially, patches. – Redbug

I'm going to try OverlayFS since AUFS was deprecated in favor of overlayfs and it should be part of the kernel already. – Armendariz

It's super weird that the reboot solved my problem. – Longfaced