Permission issue when reading container logs with fluentd in Kubernetes

Question

I am really new to kubernetes and have testing app with redis and mongodb running in GCE. I would like to grap my log files with fluentd and send them to logz:

I use the following fluentd config file. I tested a similar version on my local machine.

<source>
    @type tail
    path /var/log/containers/squidex*.log
    pos_file /var/log/squidex.log.pos
    tag squidex.logs
    format json
</source>

<match squidex.logs>
    @type copy
    <store>
        @type logzio_buffered
        endpoint_url https://listener.logz.io:8071?token=...
        output_include_time true
        output_include_tags true
        buffer_type file
        buffer_path /fluentd/log/squidex.log.buffer
        flush_interval 10s
        buffer_chunk_limit 1m
    </store>
    <store>
        @type stdout
    </store>
</match>

My kubernetes configuration is:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-logging
  labels:
    app: fluentd-logging
spec:
  template:
    metadata:
      labels:
        app: fluentd-logging
    spec:
      containers:
      - name: fluentd
        image: gcr.io/squidex-157415/squidex-fluentd:latest
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 40m
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

Almost everything works, but when I run the fluentd pods I see the following entries in the log output from these pods:

2017-04-22T09:49:22.286740784Z 2017-04-22 09:49:22 +0000 [warn]: 
/var/log/containers/squidex-282724611-3nhtw_default_squidex-ed7c437e677d3438c137cdc80110d106339999a6ba8e495a5752fe6d5da9e70d.log unreadable. 
It is excluded and would be examined next time

How can I get permissions to those log files?

Faceplate

I think that you issue is that /var/log/containers doesn't really hold the log files, but links to them. should be linked to the FS holding the containers. So you should probably mount both – Cort

Answer 1

This is not a permission issue but broken symlinks. Kubernetes is using symbolic links from /var/log/containers to /var/log/pods to /var/lib/docker/containers. You can confirm this from any node of your cluster using ls -la

Your DaemonSet configuration should include something like:

volumeMounts:
- name: varlog
  mountPath: /var/log/
  readOnly: true
  - name: varlibdockercontainers
  mountPath: /var/lib/docker/containers
  readOnly: true
[...]
volumes:
- name: varlog
  hostPath:
    path: /var/log/
- name: varlibdockercontainers
  hostPath:
    path: /var/lib/docker/containers

This way, you are mounting both the logs files directory and the symlinks of symlinks so your fluentd can read everything.

Albers

I'm mounting both directories. Still getting Permission Denied error. I also checked with /bin/sh. – Thanks