Kubernetes 137 exit code when using SecurityContext readOnlyRootFilesystem

Question

I am trying to host a web app in a container with read only file system. Whenever I try to configure the root file system as read only through the SecurityContext of the container I get the following error:

    Ports:          80/TCP, 443/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 23 Sep 2021 18:13:08 +0300
      Finished:     Thu, 23 Sep 2021 18:13:08 +0300
    Ready:          False

I've tried to achieve the same using an AppArmor profile as follows:

profile parser-profile flags=(attach_disconnected) {
  #include <abstractions/base>
   ...
  deny /** wl,
   ...

Unfortunately the result is the same.

What I assume is happening is that the container is not capable of saving the files for the web app and fails.

In my scenario, I will be running untrusted code and I must make sure that users are not allowed to access the file system.

Any ideas of what I am doing wrong and how can I achieve a read only file system?

I am using AKS and below is my deployment configuration:

apiVersion: v1
kind: Service
metadata:
  name: parser-service
spec:
  selector:
    app: parser
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: parser-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: parser
  template:
    metadata:
      labels:
        app: parser
      annotations:
        container.apparmor.security.beta.kubernetes.io/parser: localhost/parser-profile
    spec:
      containers:
      - name: parser
        image: parser.azurecr.io/parser:latest
        ports:
        - containerPort: 80
        - containerPort: 443
        resources:
          limits:
            cpu: "1.20"
        securityContext:
          readOnlyRootFilesystem: true

Edit: I also tried creating a cluster level PSP which also did not work.

Hasidism

The error that you posted - which command is it output from? Could you please run kubectl logs {pod-name} and paste logs in your question? Also I tried to replicate your issue, but I can't pull image from azurecr.io. Could you please give link to this image on Docker hub? – Kyleekylen

I have pushed the image to Docker Hub, you should be able to download it at joronator/parser:latest. However, about the logs, for some reason when I execute kubectl logs podname I get the following error Failed to create CoreCLR, HRESULT: 0x80004005. I will have to investigate a little what is happening. It is working just fine with pods from other deployments. – Hasidism

Now the image is working, thanks! Please check my answer. – Kyleekylen

Answer 1

I managed to replicate your issue and achieve read only filesystem with exception for one directory.

First, worth to note that you are using both solutions in your deployment - the AppArmor profile and SecurityContext. As AppArmor seems to be much more complex and needs configuration to be done per node I decided to use only SecurityContext as it is working fine.

I got this error that you mention in the comment:

Failed to create CoreCLR, HRESULT: 0x80004005

This error doesn't say to much, but after some testing I found that it only occurs when you are running the pod which filesytem is read only - the application tries to save files but cannot do so.

The app creates some files in the /tmp directory so the solution is to mount /tmp using Kubernetes Volumes so it will be read write. In my example I used emptyDir but you can use any other volume you want as long as it supports writing to it. The deployment configuration (you can see that I added volumeMounts and volumes and the bottom):

apiVersion: v1
kind: Service
metadata:
  name: parser-service
spec:
  selector:
    app: parser
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: parser-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: parser
  template:
    metadata:
      labels:
        app: parser
    spec:
      containers:
      - name: parser
        image: parser.azurecr.io/parser:latest
        ports:
        - containerPort: 80
        - containerPort: 443
        resources:
          limits:
            cpu: "1.20"
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /tmp
          name: temp
      volumes:
      - name: temp
        emptyDir: {}

After executing into pod I can see that pod file system is mounted as read only:

# ls   
app  bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
# touch file
touch: cannot touch 'file': Read-only file system
# mount 
overlay on / type overlay (ro,...)

By running kubectl describe pod {pod-name} I can see that /tmp directory is mounted as read write and it is using temp volume:

Mounts:
  /tmp from temp (rw)

Keep in mind if you are using other directories (for example to save files) you need to also mount them the same way as the /tmp.

Kyleekylen

That was spot on! Cheers mate! – Hasidism

Thank you, many other posts were suggesting to make use of an environment variable but this one makes more sense. – Oneman