Implications of exposing /var/lib/docker over NFS to serve hosts with limited memory

Question

What are the implications of exporting /var/lib/docker over NFS? The idea is to store the docker images in a server and export it to hosts which has limited memory to store and run containers. This would be useful to avoid having each host download and store it's own library of docker image. The hosts may make use of FS-Cache to limit the data transfer over network.

Circularize

Answer 1

The /var/lib/docker directory is designed to be exclusively accessed by a single daemon, and should never be shared with multiple daemons.

Having multiple daemons use the same /var/lib/docker can lead to many issues, and possible data corruption.

For example, the daemon keeps an in-memory state of which images are in use (by containers), and which ones not; multiple daemons using those image won't keep track of that (an image may be in use by another daemon), and remove the image while it's in use.

Docker also stores various other files in /var/lib/docker, such as a key/value store for user-defined networks, which is not designed to be accessed concurrently by multiple daemons.

Roid

Appreciate the prompt response. – Circularize

A follow up question on your response. Isn't it good to separate image layers and states in docker daemons so that multiple daemons can share images, instead of downloading it every time while launching same type of containers in a separate daemon? Put another way, does docker provide any functionality to share downloaded images across daemons running on different hosts? – Circularize

No, docker doesn't support a shared graph storage, basically for the reasons above (an image that's not used on one host, could still be used on another host); there's experimental Suppprt for third-party storage driver plug-ins though; see github.com/docker/docker/blob/…, for example, here's a ceph driver plugin; github.com/hustcat/docker-graph-driver – Roid

Thanks. I shall check it out. – Circularize