fluentd error "Systemd::JournalError: No such file or directory"

Question

I did install fluentd-kubernetes-daemonset on my kube workers on one did work without any error but on the other is throwing the following errors:

2018-12-07 03:48:33 +0000 [warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:36 +0000 [warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:39 +0000 [warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:40 +0000 [warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:45 +0000 [warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:44 +0000 [warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:48 +0000 [warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s
2018-12-07 03:48:55 +0000 [warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s

Ramp

Do you have /var/lib/dbus volume mounted into your DaemonSet so fluent can communicate with systemd? Have you tried increasing the verbosity of fluent to see what file it is trying to find? – Ralph

is not mounted... but the only file in that folder is machine-id. – Ramp

@MatthewLDaniel I did fix it I need to use fluent/fluentd-kubernetes-daemonset:v0.12-debian-elasticsearch and then set environment variable FLUENT_UID set to 0 – Ramp

I'm glad, and be aware I am pretty sure you can answer your own question, to help others know there is a solution for it – Ralph

Answer 1

If you don't setup systemd in the container, fluentd shows following messages by default configuration.

[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s

You can suppress these messages by setting disable to FLUENTD_SYSTEMD_CONF environment variable in your kubernetes configuration, e.g.:

      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: 'elasticsearch'
            - name: FLUENT_ELASTICSEARCH_PORT
              value: '9200'
            - name: FLUENT_ELASTICSEARCH_SCHEME
              value: 'http'
            - name: FLUENT_UID
              value: '0'
            - name: FLUENTD_SYSTEMD_CONF
              value: 'disable'

See Disable systemd input

Burchette