Docker: in memory file system

Question

I have a docker container which does alot of read/write to disk. I would like to test out what happens when my entire docker filesystem is in memory. I have seen some answers here that say it will not be a real performance improvement, but this is for testing.

The ideal solution I would like to test is sharing the common parts of each image and copy to your memory space when needed.

Each container files which are created during runtime should be in memory as well and separated. it shouldn't be more than 5GB fs in idle time and up to 7GB in processing time.

Simple solutions would duplicate all shared files (even those part of the OS you never use) for each container.

Mattock

Answer 1

There's no difference between the storage of the image and the base filesystem of the container, the layered FS accesses the images layers directly as a RO layer, with the container using a RW layer above to catch any changes. Therefore your goal of having the container running in memory while the Docker installation remains on disk doesn't have an easy implementation.

If you know where your RW activity is occurring (it's fairly easy to check the docker diff of a running container), the best option to me would be a tmpfs mounted at that location in your container, which is natively supported by docker (from the docker run reference):

$ docker run -d --tmpfs /run:rw,noexec,nosuid,size=65536k my_image

Outline

can i use volume on top of tmpfs? Example: docker run -d --tmpfs /run:rw,noexec,nosuid,size=65536k -v $(pwd)/database:/run/database my_image what would happen in this case? – Untouched

Obviously, According to docs.docker.com/storage/tmpfs/… : "As opposed to volumes and bind mounts, a tmpfs mount is temporary, and only persisted in the host memory. When the container stops, the tmpfs mount is removed, and files written there won’t be persisted.". This runs the entire docker file system in memory as you asked "I would like to test out what happens when my entire docker filesystem is in memory." – Diastrophism

Note that this currently works on linux only: docs.docker.com/storage/tmpfs/#limitations-of-tmpfs-mounts – Halftimbered

Answer 2

Docker stores image, container, and volume data in its directory by default. Container HDs are made of the original image and the 'container layer'.

You might be able set this up using a RAM disk. You would hard allocate some RAM, mount it, and format it with your file system of choice. Then move your docker installation to the mounted RAM disk and symlink it back to the original location.

Setting up a Ram Disk

Best way to move the Docker directory

Obviously this is only useful for testing as Docker and it's images, volumes, containers, etc would be lost on reboot.

Sward

I want the container HD to be in memory and not the docker installation on host. – Mattock

Docker stores image, container, and volume data in it's directory by default. Container HDs are made of the original image and the 'container layer'. docs.docker.com/engine/userguide/storagedriver/… – Sward

Probably better to use this, but I haven't tried: docs.docker.com/registry/storage-drivers/inmemory Also, Linux and other OSs have built-in disk caching, so your container's disk accesses might not be hitting the actual disk anyway. – Bonin

What does "HD" mean here? – Jean

@Bonin be careful with that link. The link you referenced is the storage driver for the Docker Registry, not for Docker containers! The docker registry is a service that allows you to push/pull images from it, it does not run your containers. I would instead use a tmpfs mount: docs.docker.com/storage/tmpfs – Atrophied