Getting permission denied even as root inside the docker container

Question

Connecting to running docker container as a root still gets Operation not permitted error when trying to apt-get update, yet I can still see sensitive file like /etc/passwd. Below are my configurations and also the error message from apt-get update. My host operating system is Ubuntu 18.04.3. My docker version is Docker version 19.03.5, build 633a0ea838

I create a container with the following Dockerfile

FROM python:3.8-slim-buster
RUN useradd -ms /bin/bash andrej
WORKDIR /home/andrej
COPY . /home/andrej/

RUN apt-get update && \
    apt-get install -y gcc && \
    pip install -r requirements.txt && \
    apt-get remove -y gcc && apt-get -y autoremove

RUN chown andrej:andrej pycurl && \
    chmod 0744 pycurl

USER andrej
ENTRYPOINT ["uwsgi"]
CMD ["--ini", "uwsgi.ini"]

starting it with docker compose looking like this:

version: "3.3"

services:

  andrej-cv:
    build: ./andrej_cv
    container_name: andrej-cv
    restart: always
    security_opt:
      - no-new-privileges
    expose:
      - 5000
    healthcheck:
      test: ./pycurl --host=127.0.0.1 --port=5050 --uri=/health_check
      interval: 1m30s
      timeout: 10s
      retries: 3

My docker daemon config:

{
    "icc": false,
    "userns-remap": "default",
    "log-driver": "syslog",
    "live-restore": true,
    "userland-proxy": false,
    "no-new-privileges": true
}

I connect to the container with following command (as root): docker exec -it -u root <container_hash> /bin/bash but when I try to update I got the following:

root@ed984abff684:/home/andrej# apt-get update 
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Hit:1 http://deb.debian.org/debian buster InRelease
Ign:2 http://deb.debian.org/debian buster-updates InRelease
Err:4 http://deb.debian.org/debian buster-updates Release
  Could not open file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - open (13: Permission denied) [IP: 151.101.36.204 80]
Hit:3 http://security-cdn.debian.org/debian-security buster/updates InRelease
rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chmod 0700 of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
W: Download is performed unsandboxed as root as file '/var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_Release - PrepareFiles (13: Permission denied)
E: The repository 'http://deb.debian.org/debian buster-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Problem unlinking the file /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease - PrepareFiles (13: Permission denied)

In the container /etc/subuid and /etc/subgid look like this (both):

andrej:100000:65536

On the host /etc/subuid and /etc/subgid look like this (both):

andrej:100000:65536
dockremap:165536:65536

Apparmor is running on Ubuntu host with following status (only docker-default profile):

andrej@machine:/etc/apparmor.d$ sudo aa-status 
apparmor module is loaded.
38 profiles are loaded.
36 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/8268/usr/lib/snapd/snap-confine
   /snap/core/8268/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/evince
   /usr/bin/evince-previewer
   /usr/bin/evince-previewer//sanitized_helper
   /usr/bin/evince-thumbnailer
   /usr/bin/evince//sanitized_helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/cups/backend/cups-pdf
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /usr/sbin/ippusbxd
   /usr/sbin/tcpdump
   docker-default
   libreoffice-senddoc
   libreoffice-soffice//gpg
   libreoffice-xpdfimport
   man_filter
   man_groff
   snap-update-ns.core
   snap-update-ns.gnome-calculator
   snap-update-ns.gnome-characters
   snap-update-ns.gnome-logs
   snap-update-ns.gnome-system-monitor
   snap.core.hook.configure
   snap.gnome-calculator.gnome-calculator
   snap.gnome-characters.gnome-characters
   snap.gnome-logs.gnome-logs
   snap.gnome-system-monitor.gnome-system-monitor
2 profiles are in complain mode.
   libreoffice-oopslash
   libreoffice-soffice
17 processes have profiles defined.
14 processes are in enforce mode.
   docker-default (1101) 
   docker-default (1102) 
   docker-default (1111) 
   docker-default (1600) 
   docker-default (1728) 
   docker-default (1729) 
   docker-default (1730) 
   docker-default (1731) 
   docker-default (1732) 
   docker-default (1798) 
   docker-default (1799) 
   docker-default (1800) 
   docker-default (1801) 
   docker-default (1802) 
0 processes are in complain mode.
3 processes are unconfined but have a profile defined.
   /sbin/dhclient (491) 
   /usr/sbin/cups-browsed (431) 
   /usr/sbin/cupsd (402) 

Selinux seems to be disabled as there is not /etc/selinux/config file and getenfoce and sestatus command are not available.

Also su andrej command run as root (where andrej is unprivileged user in the container) errors out with su: cannot set groups: Operation not permitted

Janise

Do you have apparmor or selinux configured on the host? – Saul

Check updated question. Also I tried to systemctl stop apparmor, systemctl disable apparmor, reboot, but the issue persists. – Janise

I have the same issue. Did you find a solution for it? – Inning

Nope. I just removed userns-remap to make it work. But was not quite happy with it as it is not solution. – Janise

May have something to do with the security option no-new-privileges. See my post below for more details. – Snug

Answer 1

Not sure about Docker, but in kubernetes in runc container for me helps:

1. Get root access to container List all containers

minikube ssh docker container ls

Connect to your container (use your container id from previous command instead of 44a7ad70d45b):

minikube ssh "docker container exec -it -u 0 44a7ad70d45b /bin/bash"

2. As root inside container:

root@mycontainer:/# apt-config dump | grep Sandbox::User
APT::Sandbox::User "_apt";

root@mycontainer:/# cat <<EOF > /etc/apt/apt.conf.d/sandbox-disable
APT::Sandbox::User "root";
EOF

Be sure, that result is valid:

root@mycontainer:/# apt-config dump | grep Sandbox::User
APT::Sandbox::User "root";

3. apt update and see

Get:1 http://archive.ubuntu.com/ubuntu hirsute InRelease [269 kB]
Get:2 http://apt.postgresql.org/pub/repos/apt hirsute-pgdg InRelease [16.7 kB]
...   

Volunteer

Thank you so much! This problem was driving me crazy. If anyone else stumbles over this, I had to update the permissions of some folders as well before it worked. They were mentioned in the apt update output. In my case chown root:root /var/cache/apt/archives/partial/ /var/lib/apt/lists/partial did the trick. – Merger

Thank you! What slightly bothers is that this problem can be reproduced by executing the following command : podman run -it --entrypoint "/usr/bin/bash" ubuntu:20.04 and entering apt update in the terminal. But only in one of the Linux machines I'm using. Actually, in one of the RedHat-hosts (another allows calling apt without tricks). Also, in Ubuntu 20.04 there's no issue. In those hosts, with no problem, there is no /etc/apt/apt.conf.d/sandbox-disable file. – Holomorphic

Answer 2

I had exatly the same issues when running Ubuntu-16.04-based container in rootless Podman with Manjaro as the host system.

TL;DR: try to rebuild the image. That helped in my case.

The issue is likely that Docker cannot map the /var/lib/apt/lists/package directory's owner (_apt) UID to host's UID namespace. This might happen if the /etc/sub{u,g}id is modified after the image is pulled/built.

This is only a guess but the reason might be that Docker performs the UID map first for the image and then modifies /etc/sub{u,g}id resulting in different UID map rules -> Docker cannot map the user inside the container.

You can verify this by running docker inspect <image name> and checking the directories in "LowerDir" part. In one of those there should exist a directory var/lib/apt/lists/package with UID outside of the range specified for dockremap in /etc/sub{u,g}id. The exact command for podman was podman image inspect <image name> --format '{{.GraphDriver.Data.LowerDir}}' but the CLI APIs of Podman and Docker should be identical so the same command should work with Docker also.

E.g. I had an entry tlammi:100000:65536 in /etc/sub{u,g}id but /var/lib/apt/lists/package was owned by UID 165538 in host side which is outside of range [100000, 100000 + 65536).

Partisan

Answer 3

We run into the same issue with rootless podman.

We changed the subuid/subgid range of the user. This means one would need to fix the files stored with the old ranges or just delete the temporary files from the container storage directory:

$ podman info|grep GraphRoot
GraphRoot: /opt/tmp/container_graphroot/.local/share/containers/storage
$ rm -rf /opt/tmp/container_graphroot/.local/share/containers/storage/*

Wilful

Answer 4

The docker container you describe is working as designed. With the default capabilities.

Fast test if you have a missing capability.

Run your container with Full/All container capabilities (--privileged)

More explanations about capabilities from Docker documentation

The Linux kernel is able to break down the privileges of the root user into distinct units referred to as capabilities. For example, the CAP_CHOWN capability is what allows the root user to make arbitrary changes to file UIDs and GIDs. The CAP_DAC_OVERRIDE capability allows the root user to bypass kernel permission checks on file read, write and execute operations. Almost all of the special powers associated with the Linux root user are broken down into individual capabilities.

This breaking down of root privileges into granular capabilities allows you to:

Remove individual capabilities from the root user account, making it less powerful/dangerous.
Add privileges to non-root users at a very granular level.

Capabilities apply to both files and threads. File capabilities allow users to execute programs with higher privileges. This is similar to the way the setuid bit works. Thread capabilities keep track of the current state of capabilities in running programs.

The Linux kernel lets you set capability bounding sets that impose limits on the capabilities that a file/thread can gain.

Docker imposes certain limitations that make working with capabilities much simpler. For example, file capabilities are stored within a file’s extended attributes, and extended attributes are stripped out when Docker images are built. This means you will not normally have to concern yourself too much with file capabilities in containers.

It is of course possible to get file capabilities into containers at runtime, however this is not recommended.

In an environment without file based capabilities, it’s not possible for applications to escalate their privileges beyond the bounding set (a set beyond which capabilities cannot grow). Docker sets the bounding set before starting a container. You can use Docker commands to add or remove capabilities to or from the bounding set.

By default, Docker drops all capabilities except those needed, using a whitelist approach.

Pavis

Answer 5

This is not an answer yet (as of 2021-05-10) but my current research on this issue so far. Hopefully, it would give others hints about where to look further. Maybe I'll come back to edit this post to make it a real answer.

As far as I see, the "issue" is caused by the use of the security option no-new-privileges. Note that it is specified in OP's docker-compose file and the Docker daemon's configuration file.

Here is its description in the Docker's doc:

--security-opt="no-new-privileges:true" Disable container processes from gaining new privileges

...

If you want to prevent your container processes from gaining additional privileges, you can execute the following command:

$ docker run --security-opt no-new-privileges -it centos bash

no-new-privileges is a flag that was added to Linux kernel since 3.5. Here is its doc:

The no_new_privs bit (since Linux 3.5) is a new, generic mechanism to make it safe for a process to modify its execution environment in a manner that persists across execve. Any task can set no_new_privs. Once the bit is set, it is inherited across fork, clone, and execve and cannot be unset. With no_new_privs set, execve() promises not to grant the privilege to do anything that could not have been done without the execve call. For example, the setuid and setgid bits will no longer change the uid or gid; file capabilities will not add to the permitted set, and LSMs will not relax constraints after execve.

Notice the last sentence of "the setuid and setgid bits will no longer change the uid or gid". This may be why you would see the following error messages:

E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)

I found an article that talks about it with examples clearly: Running Docker application containers more securely.

My current thoughts:

• I don't call the failed "apt-get update" an "issue" or a "problem" because that should be an intentional behavior for security consideration. In other words, it's a good thing.
• Because the quoted kernel doc says "Once the bit is set, it ... cannot be unset", I believe you won't be able to "fix" it in the existing containers.
• I don't think removing no-new-privileges is the right solution. At least it's not right before you fully discuss it with your team.
• Alternatively, create a container without no-new-privileges for testing purpose only.

Snug

Answer 6

NOTE: Only if you have docker and docker-compose installed If initially you had not been running as root and rebuilt the image, run a prune

docker system prune -f
docker-compose up 

This then makes sure you're running on a fresh build

Sulphuryl