FluentD log unreadable. it is excluded and would be examined next time

Question

Facing: fluentd log unreadable. it is excluded and would be examined next time

I have a simple configuration for fluentD daemon set running in kubernetes setup.

Fluentd version: fluentd-0.12.43

Below is my configuration.

  <source>
    @type tail
    path /var/log/containers/sample*.log
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag sample.*
    format json
    read_from_head true
  </source>
  <match sample.**>
    @type forward
    heartbeat_type tcp
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s
    <server>
      name worker-node2
      host 10.32.0.15
      port 24224
      weight 60
    </server>
  </match>

Getting below warning and NO logs are forwarded

2018-08-03 06:36:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5d6328e7d.log unreadable. It is excluded and would be examined next time.

2018-08-03 06:37:53 +0000 [warn]: /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5bc89ab24.log unreadable. It is excluded and would be examined next time.

Permission for log file:

[root@k8s-master fluentd-daemonset]# ls -lrt **/var/log/containers/**

**lrwxrwxrwx** Jun 25 06:25 sample-77g68_kube-system_kube-proxy-9f3c3951c32ee.log 
-> /var/log/pods/aa1f8d5b-746f-11e8-95c0-005056b9ff3a/sample/7.log

YAML file for daemon set have mount instructions:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging1
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    -----
    -----
    -----

        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc/ 
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlogcontainers
          mountPath: /var/log/containers
          readOnly: true
        - name: varlibdocker
          mountPath: /var/lib/docker
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlogcontainers
        hostPath:
          path: /var/log/containers
      - name: varlibdocker
        hostPath:
          path: /var/lib/docker
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers 

Getting no clue even when permission is correct, fluentD version is correct, Mount instruction are their in kubernetes daemonset, why I am getting this warning.

Revanchism

Answer 1

I faced a similar issue. So, what actually happens is -

1. Fluentd creates a symbolic links in /var/log/containers/ which are further a symbolic link of files that are in /var/log/pods/ like -

root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log 
lrwxrwxrwx. 1 root root 83 Oct 30 07:42 /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log -> /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1628.log

2. /var/log/pods are symbolic links of the log files mounted at host. In my setup I am using /data/ directory of host/node to store docker data.

root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log 
lrwxrwxrwx. 1 root root 162 Oct 30 07:47 /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log -> /data/docker/containers/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b-json.log

So, in my deployment.yaml I had to mount /data/docker/containers rather /var/lib/containers/ to solve the issue i.e

        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /data/docker/containers
          name: datadockercontainers
          readOnly: true
        - mountPath: /fluentd/etc
          name: config-path

Alissaalistair

For us, we traced the symbolic links to /u01/data. This need to be mounted to solve the issue. – Michealmicheil

Answer 2

May colachg suggestion help you:

I think that kubelet create some symbolic links in '/var/log/containers'(just links not real file), so you must mount both links and real files or only mount real file with right fluentd.conf.

Erastianism

Answer 3

As you defined /var/log in the list, the others /var/log/... are duplicated.

Remove /var/log

Check with kubectl describe pod fluentd-... whether all volumes were mounted properly.

Spectacular

All the volumes are mounted. Mounts: /fluentd/etc/ from fluentd-config (rw) /var/lib/docker from varlibdocker (ro) /var/lib/docker/containers from varlibdockercontainers (ro) /var/log from varlog (ro) /var/log/containers from varlogcontainers (ro) /var/log/pods from varlogpods (ro) /var/run/secrets/kubernetes.io/serviceaccount from default-token-54bgv (ro) – Revanchism

Try to remove /var/log – Spectacular

I have tried all these. Like putting /var/log and removing it. Still not working. – Revanchism

Answer 4

To add a securityContext field under spec.containers, you can use the following YAML code:

spec:
  containers:
  - name: fluentd
    image: fluent/fluentd:v1.16.3-debian-amd64-1.0
    securityContext:    # this
      runAsUser: 0
    # Rest of the configuration...

By adding the securityContext field with runAsUser: 0, you are setting the user ID for the fluentd container to 0, which is the root user. This allows fluentd to run as the root user within the container.

Uncovenanted

Answer 5

We need to set the below environment variable: FLUENT_UID to 0

Revanchism

any solution for this ? – Rowney

We need to set the below environment variable: FLUENT_UID to 0 – Revanchism

Hi Hemant, where did you set this ENV variable? in values.yaml or configmap? – Gateshead

Answer 6

/var/log/containers/*.log  unreadable. 

The most direct way is to change mode:

chmod 777 /var/log/containers/*.log

but the best way is: change fluent user to root (set FLUENT_UID environment variable to 0 in your docker/kubernetes configuration);

add --env FLUENT_UID=0 to docker command, for example:

docker run -it -d   -p 24224:24224   -v /path/to/conf:/fluentd/etc   -v /var:/var --env FLUENT_UID=0 fluent/fluentd:latest

or add to Kubernetes yaml file:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  # namespace: default
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.4-debian-elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch.logging"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENT_UID  # change this place
            value: "0"

Wigwam

Answer 7

You have to set this env vars:

- name: FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
  value: '["/var/log/containers/fluentd-*"]'
- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: "/^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/"

Macintosh