why I am getting Read only file system error from Nginx?

Question

Dear K8S community Team,

I am getting this error message from nginx when I deploy my application pod. My application an angular6 app is hosted inside an nginx server, which is deployed as a docker container inside EKS.

I have my application configured as a “read-only container filesystem”, but I am using “ephemeral mounted” volume of type “emptyDir” in combination with a read-only filesystem.

So I am not sure the reason of this following error:

2019/04/02 14:11:29 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)

My deployment.yaml is:

...
 spec:
      volumes:
        - name: tmp-volume
          emptyDir: {}
        # Pod Security Context
      securityContext:
        fsGroup: 2000
      containers:
      - name: {{ .Chart.Name }}
        volumeMounts:
        - mountPath: /tmp
          name: tmp-volume
        image: "{{ .Values.image.name }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        securityContext:
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
        securityContext:
          readOnlyRootFilesystem: true
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
...

nginx.conf is:

...
http {

include           /etc/nginx/mime.types;
  default_type      application/octet-stream;

  # Turn off the bloody buffering to temp files
  proxy_buffering off;

  sendfile          off;
  keepalive_timeout 120;

  server_names_hash_bucket_size 128;

  # These two should be the same or nginx will start writing 
  #  large request bodies to temp files
  client_body_buffer_size 10m;
  client_max_body_size    10m;
...

Accessory

Hi, you are mounting it to /tmp but creating the file at /var/cache/nginx/client_temp. These are 2 distinct location. /tmp is coming from emptydir but other is part of the container file system which is readonly – Reeding

you are right! Now i am redirecting nginx to create files here at my mounted volume: nginx.conf code .. http { client_body_temp_path /tmp 1 2; proxy_temp_path /tmp 1 2; fastcgi_temp_path /tmp 1 2; uwsgi_temp_path /tmp 1 2; scgi_temp_path /tmp 1 2; ... server { listen 0.0.0.0:80; code but now getting this error: 2019/04/02 15:22:43 [emerg] 1#1: bind() to 0.0.0.0:80 failed (13: Permission denied) nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied) – Accessory

I had to add as well proxy_cache_path /tmp/cache levels=1:2 keys_zone=one:10m; in default.conf before the server declaration. – Warms

Answer 1

Seems like your nginx is not running as root user.

Since release 1.12.1-r2, nginx daemon is being run as user 1001.

1.12.1-r2

The nginx container has been migrated to a non-root container approach. Previously the container run as root user and the nginx daemon was started as nginx user. From now own, both the container and the nginx daemon run as user 1001. As a consequence, the configuration files are writable by the user running the nginx process.

This is why you are unable to bind on port 80, it's necessary to use port > 1000.

You should use:

  ports:
   - '80:8080'
   - '443:8443'

and edit the nginx.conf so it listens on port 8080:

server {
        listen 0.0.0.0:8080;
        ...

Or run nginx as root: command: [ "/bin/bash", "-c", "sudo nginx -g 'daemon off;'" ]

Brochu

Answer 2

As already stated by Crou, the nginx image maintainers switched to a non-root-user-approach.

This has two implications:

1. Your nginx process might not be able to bind all network sockets.
2. Your nginx process might not be able to read all file system locations.

You can try to change the ports as described by Crou (nginx.conf and deployment.yaml). Even with the NET_BIND_SERVICE capability added to the container, this does not neccessarily mean that the nginx process gets this capability. You can try to add the capability with

$ sudo setcap 'cap_net_bind+p' $(which nginx)

as a RUN instruction in your Dockerfile. However it is usually simpler to just change the listening port.

For the filesystem, please note that /var/cache/nginx/ is not mounted as a volume and thus belongs to the RootFS which is mounted as read only. The simplest way to solve this, is to add a second epheremal emptyDir for /var/cache/nginx/ in the volumes section. Please make sure, that the nginx user has the file system permissions to read and write this directory. This is usually already taken care of by the docker image maintainers as long as you stay with the default locations.

I recommend you to not switch back to running nginx as root as this might expose you to security vulnerabilities.

Campball