Is it possible to run gcsfuse without privileged mode inside GCP kubernetes?

Question

Following this guide,I am trying to run gcsfuse inside a pod in GKE. Below is the deployment manifest that I am using:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gcsfuse-test
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: gcsfuse-test
    spec:
      containers:
        - name: gcsfuse-test
          image: gcr.io/project123/gcs-test-fuse:latest
          securityContext:
            privileged: true
            capabilities:
              add:
                - SYS_ADMIN
          lifecycle:
            postStart:
              exec:
                command: ["mkdir", "/mnt"]
                command: ["gcsfuse", "-o", "nonempty", "cloudsql-p8p", "/mnt"]
            preStop:
              exec:
                command: ["fusermount", "-u", "/mnt"]

However, I would like to run gcsfuse without the privileged mode inside my GKE Cluster. I think (because of questions like these on SO) it is possible to run the docker image with certain flags and there will be no need to run it in privileged mode.

Is there any way in GKE to run gcsfuse without running the container in privileged mode?

Doodlebug

Answer 1

Privileged mode means you have all the capabilities enabled, see https://stackoverflow.com/a/36441605. So adding CAP_SYS_ADMIN looks redundant here in your example.

You can either give all the privileges or do something more fine-grained by mounting /dev/fuse and giving only SYS_ADMIN capability (which remains an important permission).

I think we can rephrase the question as : Can we run GCSFuse without the capability SYS_ADMIN ?

Actually it does not look feasible, you can find the related docker issue here : https://github.com/docker/for-linux/issues/321.

For most of projects it won't be a hard no-go. You may want to act in regard of your threat model and decide whether or not it may be a security risk for your production.

Regent

So, what is the minimum permission a docker container needs to mount external volumes (like gcsfuse)? I tried removing privileged: true and just keeping the SYS_ADMIN part, it does not work. – Doodlebug

The issue likely comes from the apparmor profile. Set unconfined as apparmor profile to the fuse container through annotations : kubernetes.io/docs/tutorials/clusters/apparmor/#api-reference – Regent

So what does this answer actuallyl means ? How can we get gcsfuse work with gke ? – Headdress

Answer 2

Edit Apr 26, 2022: for a further developed repo derived from this answer, see https://github.com/samos123/gke-gcs-fuse-unprivileged

Now it finally is possible to mount devices without privileged: true or CAP_SYS_ADMIN!

What you need is

• A Kubelet device manager which allows containers to have direct access to host devices in a secure way. The device manager explicitly given devices to be available via the Kubelet Device API. I used this hidden gem: https://gitlab.com/arm-research/smarter/smarter-device-manager.

• Define list of devices provided by the Device Manager - add /dev/YOUR_DEVICE_NAME into this list, see example below.

• Request a device via the Device Manager in the pod spec resources.requests.smarter-devices/YOUR_DEVICE_NAME: 1

I spent quite some time figuring this out so I hope sharing the information here will help someone else from the exploration.

I wrote my details findings in the Kubernetes Github issue about /dev/fuse. See an example setup in this comment and more technical details above that one.

Examples from the comment linked above:

Allow FUSE devices via Device Manager:

apiVersion: v1
kind: ConfigMap
metadata:
  name: smarter-device-manager
  namespace: device-manager
data:
  conf.yaml: |
    - devicematch: ^fuse$
      nummaxdevices: 20

Request /dev/fuse via Device Manager:

# Pod spec: 
          resources:
            limits:
              smarter-devices/fuse: 1
              memory: 512Mi
            requests:
              smarter-devices/fuse: 1
              cpu: 10m
              memory: 50Mi

Device Manager as a DaemonSet:

# https://gitlab.com/arm-research/smarter/smarter-device-manager/-/blob/master/smarter-device-manager-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: smarter-device-manager
  namespace: device-manager
  labels:
    name: smarter-device-manager
    role: agent
spec:
  selector:
    matchLabels:
      name: smarter-device-manager
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: smarter-device-manager
      annotations:
        node.kubernetes.io/bootstrap-checkpoint: "true"
    spec:
      ## kubectl label node pike5 smarter-device-manager=enabled
      # nodeSelector:
      #   smarter-device-manager : enabled
      priorityClassName: "system-node-critical"
      hostname: smarter-device-management
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: smarter-device-manager
        image: registry.gitlab.com/arm-research/smarter/smarter-device-manager:v1.1.2
        imagePullPolicy: IfNotPresent
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        resources:
          limits:
            cpu: 100m
            memory: 15Mi
          requests:
            cpu: 10m
            memory: 15Mi
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
          - name: dev-dir
            mountPath: /dev
          - name: sys-dir
            mountPath: /sys
          - name: config
            mountPath: /root/config
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: dev-dir
          hostPath:
            path: /dev
        - name: sys-dir
          hostPath:
            path: /sys
        - name: config
          configMap:
            name: smarter-device-manager

Woollen

I’m out of my depth here, but my interpretation of that thread and subsequent comments you made there is that some elevated permissions are still required anyway (namely CAP_SYS_ADMIN). Someone posted a link to this article which goes into great detail explaining why. So, it looks possible in GKE which is great, but doesn’t seem as secure as this answer initially suggested (and probably isn’t compatible still with GKE Autopilot for this and other reasons, how I ended up here 😊). – Strophic

@chunk_split CAP_SYS_ADMIN is not required precisely for the reason that the Kubernetes Device Manager API is used. A seccomp profile can be defined to grant exactly the permissions required for mounting, without the full CAP_SYS_ADMIN. – Woollen

This is a spectacular approach @PetrusRepo, I appreciate you sharing (afaict this is by far the best answer in early 2022), we might be able to format your answer a little better to be more accessible to K8S novices, this is such a great setup, would you mind if I edited your answer a bit to make it a little easier to follow? – Tetrapod

Hi @PetrusRepo, your own comment in the github thread is the missing piece, I think the SO answer makes more sense with your wonderful examples, so I went ahead and added them "in your words". Thank you again for sharing your discovery here, this is a problem I have thought a lot about too, and its wonderful to see a breakthrough, I'll be happy to see it more widely known! 🙏🏽 – Tetrapod