How to configure fluentd daemonset for RBAC

Question

Issue

The fluentd daemonset manifest in Kubernetes Logging with Fluentd will cause an authorization error if RBAC is enabled.

$ kubectl logs fluentd-4nzv7 -n kube-system
2018-01-06 11:28:10 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2018-01-06 11:28:10 +0000 [info]: starting fluentd-0.12.33
2018-01-06 11:28:10 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.10.0'
2018-01-06 11:28:10 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.29.0'
2018-01-06 11:28:10 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.9.1'
2018-01-06 11:28:10 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5'
2018-01-06 11:28:10 +0000 [info]: gem 'fluentd' version '0.12.33'
2018-01-06 11:28:10 +0000 [info]: adding match pattern="fluent.**" type="null"
2018-01-06 11:28:10 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2018-01-06 11:28:11 +0000 [info]: adding match pattern="**" type="elasticsearch"
2018-01-06 11:28:11 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: pods is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list pods at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"pods is forbidden: User \\\"system:serviceaccount:kube-system:default\\\" cannot list pods at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"pods\"},\"code\":403}\n)"
2018-01-06 11:28:11 +0000 [info]: process finished code=256
2018-01-06 11:28:11 +0000 [warn]: process died within 1 second. exit.

Beatification

Answer 1

When you are defining your daemonset you can also define your RBAC.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluentd-service-account
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-service-account
subjects:
- kind: ServiceAccount
  name: fluentd-service-account
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-service-account
  namespace: kube-system
rules:
  - apiGroups: ["*"]
    resources:
      - pods
      - namespaces
    verbs:
      - get
      - watch
      - list

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-service-account
  namespace: kube-system

Source.

Unilocular

Along with above changes add below code to daemonset specs serviceAccount: fluentd-service-account serviceAccountName: fluentd-service-account – Fairweather

@j.davies, I need configure FluentD to send logs from my AWS-EKS nodes to AWS managed OpenSearch dashboard. So elasticsearch is needed? FluentD as DaemonSet installed on each of my EKS node. – Hydride

Answer 2

Get "403 Forbidden" message when running the pod

The links shows the solution.

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: system

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd
  namespace: system
data:
  fluent.conf: |
    @include kubernetes.conf

    <match **>
       type elasticsearch
       log_level info
       include_tag_key true
       host elastic.system.svc.cluster.local
       port 9200
       user elastic
       password <...>
       logstash_format true
       buffer_chunk_limit 2M
       buffer_queue_limit 32
       flush_interval 5s
       max_retry_wait 30
       disable_retry_limit
       num_threads 8
    </match>

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:elasticsearch
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
            - name: config
              mountPath: /fluentd/etc/fluent.conf
              subPath: fluent.conf
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: config
          configMap:
            name: fluentd

Beatification