Permission denied when running `mkdir` inside of a Docker container

Question

I am using Docker Compose to run several containers, including one with a Postgres image. I am attempting to add a volume to that container to persist my data across container builds. However, I am receiving an error when it tries to create a directory for this volume within the container.

I run:

docker-compose build

then

docker-compose up

And I receive the following error:

ERROR: for cxbenchmark_db_1 Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data\\" to rootfs \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged\\" at \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data\\" caused \\"mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied\\"\""

My full docker-compose.yml looks like this (note the service called db where the volume is defined):

version: '3'  

services:  
  nginx:
    image: nginx:latest
    ports:
      - 80:8000
    volumes:
      - ./src:/src
      - ./config/nginx:/etc/nginx/conf.d
      - ./src/static:/static
    depends_on:
      - web

  web:
    build: .
    command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn cx_benchmark.wsgi -b 0.0.0.0:8000"
    depends_on:
      - db
    volumes:
      - ./src:/src
      - ./src/static:/static
    expose:
      - 8000

  db:
    image: postgres:latest
    volumes:
      - /private/var/lib/postgresql:/var/lib/postgresql
    ports:
      - 5432:5432

Any ideas for how to solve?

Birthstone

Answer 1

The error you are seeing is not a problem (necessarily) with the explicit volume bind mount in your compose file, but rather with the VOLUME declaration in the main postgres official Docker image Dockerfile:

VOLUME /var/lib/postgresql/data

Since you haven't provided a mount-point for this directory (but rather the parent), the docker engine is creating a local volume and then trying to mount that volume into your already bind-mounted location and getting a permissions error.

For clarity, here is the volume the docker engine created for you:

/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data

And here is the directory location at which it is trying to bind mount that dir; on top of your bind mount from /private/var/lib/postgresql:

mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied

Now, I think the reason this is failing is that you may have turned on user namespaces in your Docker engine ("userns-remap" flag/setting) such that the container doesn't have permissions to create a directory in that root-owned location on your host. Barring that, the only other option is that the postgres container is starting as a non-root user, but I don't see anything in your compose file or the official Dockerfile for the latest release that uses the USER directive.

As an aside, since you are ending up with double-volumes because your bind mount doesn't match the VOLUME specifier in the postgres Dockerfile, you could change your compose file to mount to /var/lib/postgresql/data and get around that extra volume being created. Especially if you expect your DB data to end up in /private/var/lib/postgresql, as it may be surprising to find it isn't there, but rather in the /var/lib/docker/volumes/.. location.

Eschew

Thank you for the insight! I definitely don't want those double volumes. I changed the volumes entry in the db service of my docker-compose file to: /private/var/lib/postgresql/data:/var/lib/postgresql/data But now I get the following error when the postgres container tries to change the permissions on that directory: db_1 | chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted Any ideas for how I can fix that? – Birthstone

Yes, so the entrypoint script for the official image also tries to chown the data directory, and that is now trying to change ownership of the host-mounted directory. This is why I'm curious if your Docker engine is running with user namespaces enabled as normally this action would work for the root user. However, the paths shown above in /var/lib/docker make me think this isn't the case. What are the permission bits on /private/var/lib/postgresql/data and who owns it? Is this example using "Docker for Mac"? – Eschew

Thank you @PhilE. I had the exact same problem and I was struggling with it for weeks. Your solution worked perfectly. :) – Counterstamp

Hi, I was getting the same problem, I solved in other manner, but, I can't understand all your solution. What was really your solution? change permission to user of executes docker? – Tyratyrannical

@PhilE Was this resolved by anyone ? I am stuck with the same issue. Can we change the docker engine user ? Changing the flag dint work for me. I am on macOs. – Coronado

Had this problem as well when selinux is enabled - you can test by disabling it with setenforce 0 – Fauteuil

I suppose this gave me a little insight, but I'm also stuck; I can't seem to find an actual solution here. – Iorgos

Answer 2

I created a Postgres stateful set using this command it is working and persisting data when I delete the Postgres container and re-initiate it. under the env:

• name: PGDATA

  value: /var/lib/postgresql/data

and under the volumeMounts option:

• name: nameOfPVC

  mountPath: /var/lib/postgresql/data

  subPath: data

this should work fine if you are trying to work with statefulset.

Frausto

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Fluorene