Move docker volume to different partition

Question

I have a server where I run some containers with volumes. All my volumes are in /var/lib/docker/volumes/ because docker is managing it. I use docker-compose to start my containers.

Recently, I tried to stop one of my container but it was impossible :

$ docker-compose down
[17849] INTERNAL ERROR: cannot create temporary directory!

So, I checked how the data is mounted on the server :

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7,8G     0  7,8G   0% /dev
tmpfs           1,6G  1,9M  1,6G   1% /run
/dev/md3         20G   19G     0 100% /
tmpfs           7,9G     0  7,9G   0% /dev/shm
tmpfs           5,0M     0  5,0M   0% /run/lock
tmpfs           7,9G     0  7,9G   0% /sys/fs/cgroup
/dev/md2        487M  147M  311M  33% /boot
/dev/md4        1,8T  1,7G  1,7T   1% /home
tmpfs           1,6G     0  1,6G   0% /run/user/1000

As you can see, the / is only 20Go, so it is full and I can't stop my containers using docker-compose.

My questions are :

• There is a simple solution to increase the available space in the /, using /dev/md4 ?

• Or can I move volumes to another place without losing data ?

Launch

You should probably mount a separate filesystem at /var/lib/docker so that your container storage doesn't fill up your root filesystem. This will require stopping docker and moving files around. – Leitman

Solved by moving /var to another partition like this : serverfault.com/questions/429937/… – Launch

Answer 1

This part of the Docker Daemon is configurable. Best practices would have you change the data folder; this can be done with OS-level Linux commands like a symlink... I would say it's better to actually configure the Docker Daemon to store the data elsewhere!

You can do that by editing the Docker command line (e.g. the systemd script that starts the Docker daemon), or change /etc/docker/daemon.json.

The file should have this content:

{
  "data-root": "/path/to/your/docker"
}

If you add a new hard drive, partition, or mount point you can add it here and docker will store its data there.

Eyeful

Answer 2

To move an existing docker data folder, do the following:

1. Stop the docker daemon:

service docker stop

2. Create/Edit the /etc/docker/daemon.json configuration file with location of the new data directory:

{
    "data-root": "/new/path"
}

3. Copy docker files to new location:

rsync -aP /var/lib/docker/ /new/path

4. Remove old directory (rename it to be safe)

mv /var/lib/docker /var/lib/docker.old

5. Create symlink:

ln -s /new/path /var/lib/docker

Note: If I don't create the symlink, I get this error:

Error response from daemon: error evaluating symlinks from mount source "/var/lib/docker/...": lstat /var/lib/docker: no such file or directory

Here it says that the error can be fixed by changing the path in the /new/path/containers/*/config.v2.json files, but that didn't work for me, the original /var/lib/docker path reappeared in those files.

6. Start daemon

service docker start

7. Test

docker run --rm hello-world

Should see something like this:

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

8. Cleanup

rm -rf /var/lib/docker.old

Caceres

this was extremely useful, thanks! In my case, i didn't need step 5 - it worked without a symlink – Lexicostatistics

Can confirm that I did not need to run step 5 on Debian 12 with docker version: 26.1.3, build b72abbb – Electrolytic

Answer 3

I landed here as I had the very same issue. Even though some sources suggest you could do it with a symbolic link this will cause all kinds of issues.

Depending on the OS and Docker version I had malformed images, weird errors or the docker-daemon refused to start.

Here is a solution, but it seems it varies a little from version to version. For me the solution was:

Open

/lib/systemd/system/docker.service

And change this line

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

to:

ExecStart=/usr/bin/dockerd -g /mnt/WHATEVERYOUR/PARTITIONIS/docker --containerd=/run/containerd/containerd.sock

Annalee

what is /mnt ? Can I directly write my own directory? – Erosive

Of course, you can. Replace /mnt/WHATEVERYOUR/PARTITIONIS/docker with your new path – Sargeant

I find ZaxLofful's answer better as it uses a user configuration instead of changing a system script that might be accidentally updated and is harder to find. – Outwardbound

Correct, do not use this answer. Use @Eyeful answer. When you update Docker you can run into problems. – Cogon

Answer 4

Do a bind mount.

For example, moving /docker/volumes to /mnt/large.

Append line into /etc/fstab.

/mnt/large /docker/volumes none bind 0 0

And then.

mv /docker/volumes/* /mnt/large/
mount /docker/volumes

Do not forget chown and chmod of /mnt/large first, if you are using non-root docker.

Littleton

this is seems the most permanent solution. I like this one most. just would replace the /mnt/large with UUID. – Lymphangial

@Lymphangial Oh, my example using /mnt seems misleading. The source can be arbitrary path, like /data/service-user/docker-vol, if the admin defines scopes for apps. – Littleton

do i need to restart the docker daemon for this? I guess so, right? – Ankus

@Ankus without restart, the daemon may still use old place. it is safer to stop daemon before mounting. – Littleton

Answer 5

I solved it creating a symbolic link to a partition with bigger size:

ln -s /scratch/docker_meta /var/lib/docker

/scratch/docker_meta is the folder that I have in a bigger partition.

Evangelina

While I do agree, that this works; I also agree with it being duct tape job....Use the other answers for a long term fix. – Eyeful

Definitely DO NOT use a symlink. You will run into multiple problems. – Cogon

What are the problems of using symlinks @Cogon ? – Evangelina

@Evangelina I encourage you to take a moment & Google Docker Symlinks. You will find that there are troves of articles on why you should 'not' use symlinks with Docker. Also, I also encourage you to review the open issues with Docker regarding symlinks. Using comments does not fortitude enough characters to write an appropriate answer. I use Docker daily & knew better, yet I still tried to use symlinks. And after just two days, I deeply regretted that decision. Here are two links to start with; https://mcmap.net/q/269035/-docker-and-symlinks & github.com/moby/moby/issues/1676 – Cogon