docker-compose tmpfs not working

Question

I have a docker-compose file that I'm trying to secure by making the root volumes of the containers it creates read-only.

Relevant parts of docker-compose.yml:

version: '2'
services:
  mysql:
    image: mariadb:10.1
    read_only: true
    tmpfs:
    - /var/run/mysqld:uid=999,gid=999
    - /tmp
    volumes:
    - mysql:/var/lib/mysql
    restart: always
volumes:
  mysql:

Trouble is, the tmpfs isn't being created. If I run an instance of the container using docker-compose run --rm mysql /bin/bash, the /var/run/mysqld directory is still read-only despite the tmpfs entry, and any attempt to touch /var/run/mysqld/foo fails. Since this is where MySQL puts its socket and pid file, this causes the whole thing to fail. I'm not sure why the tmpfs entry isn't working in this case.

mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] mysqld (mysqld 10.1.21-MariaDB-1~jessie) starting as process 1 ...
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: The InnoDB memory heap is disabled
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Compressed tables use zlib 1.2.8
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Using Linux native AIO
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Using SSE crc32 instructions
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Completed initialization of buffer pool
mysql_1    | 2017-01-27 20:53:45 140515784030144 [Note] InnoDB: Highest supported file format is Barracuda.
mysql_1    | 2017-01-27 20:53:48 140515784030144 [Note] InnoDB: 128 rollback segment(s) are active.
mysql_1    | 2017-01-27 20:53:48 140515784030144 [Note] InnoDB: Waiting for purge to start
mysql_1    | 2017-01-27 20:53:48 140515784030144 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 239403989
mysql_1    | 2017-01-27 20:53:48 140515005662976 [Note] InnoDB: Dumping buffer pool(s) not yet started
mysql_1    | 2017-01-27 20:53:48 140515784030144 [Note] Plugin 'FEEDBACK' is disabled.
mysql_1    | 2017-01-27 20:53:49 140515784030144 [Note] Server socket created on IP: '::'.
mysql_1    | 2017-01-27 20:53:49 140515784030144 [ERROR] Can't start server : Bind on unix socket: Read-only file system
mysql_1    | 2017-01-27 20:53:49 140515784030144 [ERROR] Do you already have another mysqld server running on socket: /var/run/mysqld/mysqld.sock ?
mysql_1    | 2017-01-27 20:53:49 140515784030144 [ERROR] Aborting

I can verify the permissions on the directory are correct (and that the UID of the mysql user is 999):

$ ls -la /var/run/mysqld 
total 8
drwxrwxrwx 2 mysql mysql 4096 Jan 17 22:14 .
drwxr-xr-x 4 root  root  4096 Jan 18 22:55 ..

But I still cannot:

$ touch /var/run/mysqld/foo
touch: cannot touch '/var/run/mysqld/foo': Read-only file system

Even if I run as root.

Any ideas what I'm doing wrong?

As an aside, the /tmp filesystem works fine.

Crapulous

Are you using docker swarm? docs.docker.com/compose/compose-file/#tmpfs says "Note: This option is ignored when deploying a stack in swarm mode with a (version 3) Compose file." – Arrear

Nope. Straight up docker-compose. – Crapulous

Me too. (filler) – Kiyokokiyoshi

Answer 1

I have been doing some testing in this regards, it looks like the /var/run directory is special in docker.

Here is some sample config and output:

  ubuntu:
    image: ubuntu
    command: "bash -c 'mount'"
    tmpfs:
      - /var/run
      - /var/cache

Running docker-compose up ubuntu shows what is mounted. Can see /var/cache is mounted but /var/run isn't.

...
ubuntu_1           | tmpfs on /var/cache type tmpfs (rw,nosuid,nodev,noexec,relatime)
...

If you use docker-compose run ubuntu bash you can see it's also mounted there but not /var/run.

The reason is that /var/run is normally a symlink to /run and hence you creating /var/run/mysql as a tmpfs doesn't work.

It will work if you change it to /run/mysql, but /run is normally mounted as tmpfs anyway so you might as well just make /run a tmpfs. Like so:

  ubuntu:
    image: ubuntu
    command: "bash -c 'mount'"
    tmpfs:
      - /run
      - /var/cache

Note: I'd like to amend my answer and show the way to do it using volumes:

services:
  ubuntu:
    image: ubuntu
    command: "bash -c 'mount'"
    volumes:
      - cache_vol:/var/cache
      - run_vol:/run

volumes:
  run_vol:
    driver_opts:
      type: tmpfs
      device: tmpfs
  cache_vol:
    driver_opts:
      type: tmpfs
      device: tmpfs

This also allows you to share the tmpfs mounts if needed.

Io

According to documentation tmpfs mounts cannot be shared among containers: docs.docker.com/engine/admin/volumes/tmpfs/… – Kristalkristan

Yes the tmpfs mount cannot, but above is a volume that is using the tmpfs type. Which is not a tmpfs mount but a volume mount using the tmpfs device. Which in theory can be shared like any other volume. – Io

Has Docker's behaviour changed recently in this regard, as I used be able to tmpfs mount /var/run/ and this has only stopped working today during a reboot on a frequently updated CoreOS machine. – Dig

Bardi it's probably because /var/run is a symlink – Io

@Io Tmpfs's lifespan is normally 1:1 with the container's lifespan. If multiple containers share a volume using the tmpfs type and device, then what becomes of the lifecycle of the data in the volume? – Iinden

Good question Jonathan, since it's a volume mount of the tmpfs type it would persist. Like other volumes do – Io

Answer 2

it some images like alpine the directory /var/run is just a link to /run you can verify that using

$ docker run --rm -ti mariadb:10.1 ls -lh /var/run
lrwxrwxrwx 1 root root 4 Aug  7 13:02 /var/run -> /run

which means that /var/run/mysqld is actually /run/mysqld

your updated docker-compose.yml is

version: '2'
services:
  mysql:
    image: mariadb:10.1
    read_only: true
    tmpfs:
    - /run/mysqld:uid=999,gid=999
    - /tmp
    volumes:
    - mysql:/var/lib/mysql
    restart: always
volumes:
  mysql:

in that case just make your tmpfs point to /run

it looks like the /var/run directory is special in docker.

no, it's just because it's a link

Working

Thanks, :uid=999,gid=999 fixed my issue with non-writable directory. – Vyborg