FluentBit not reading logs from shared PVC

Question

I have a PersistentVolumeClaims(PVC) with ReadWriteMany (RWX) access mode. This PVC is clamed by 2 pods. In the second pod I have FluentBit reading the logs stored by the application pod in the shared PVC.
The issue is FluentBit recognizes the files. FluentBit prints to the log that it found a new file. But it does not read the content of new file as and how logs are inserted into the file.
I am using OpenShift 3.11, if that makes any difference. How do I share the PVC between pods?

Logs from fluentBit console.

Fluent Bit v1.7.9
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io 

[2021/07/21 08:20:09] [ info] [engine] started (pid=1)
[2021/07/21 08:20:09] [ info] [storage] version=1.1.1, initializing...
[2021/07/21 08:20:09] [ info] [storage] in-memory
[2021/07/21 08:20:09] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/07/21 08:20:09] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/07/21 08:20:09] [ info] [sp] stream processor started
[2021/07/21 08:20:09] [ info] [input:tail:tail.0] inotify_fs_add(): inode=97 watch_fd=1 name=/var/log/log-logging-poc-standalone-app-54d8467d6f-f9j5c-2021072011.json
[2021/07/21 08:20:09] [ info] [input:tail:tail.0] inotify_fs_add(): inode=54346 watch_fd=1 name=/var/log/log-logging-poc-standalone-app-54d8467d6f-f9j5c-2021072012.json
[2021/07/21 08:20:09] [ info] [input:tail:tail.0] inotify_fs_add(): inode=43255 watch_fd=1 name=/var/log/log-logging-poc-standalone-app-54d8467d6f-f9j5c-2021072013.json

First - Application pod claims this to store the application logs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logging-poc-standalone-app
spec:
  selector:
    matchLabels:
      app: logging-poc-standalone-app
  template:
    metadata:
      labels:
        app: logging-poc-standalone-app
    spec:
      containers:
      - name: logging-poc-standalone-app
        image: loggingpoc:dev
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080
        volumeMounts:
            - name: log-volume
              mountPath: /var/log/
      securityContext:
        supplementalGroups: [100003] 
        privileged: false
      volumes:
        - name: log-volume
          persistentVolumeClaim:
            claimName: data-3 

 

Second - FluentBit pod claims the same PVC in readOnly mode.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: "fluent/fluent-bit:1.7-debug"
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: "200Mi"
            cpu: "500m"
        ports:
        - name: matrices
          containerPort: 2020
          protocol: TCP
        volumeMounts:
          - name: log-volume
            mountPath: /var/log/
          - name: config-volume
            mountPath: /fluent-bit/etc/ 
      securityContext:
        supplementalGroups: [100003] 
        privileged: false
      volumes:
        - name: log-volume
          persistentVolumeClaim:
            claimName: data-3
            readOnly: true
        - name: config-volume
          configMap:
            name: flb-sidecar

Following is the fluentbit configuration -

data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name                tail
        Tag                 demo.wen.api
        Parser              json
        Path                /var/log/log-*.json
        Mem_Buf_Limit       10MB
        Skip_Long_Lines     On
        Refresh_Interval    10

    [OUTPUT]
        Name stdout
        Match *

  parsers.conf: |
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

Grandparent

Maybe try to rule out fluent-bit. Does just running tail Linux command in the side car output what's expected? Are your log lines separated by \n (new line characters)? – Dismiss

Here fluentBit is not a sidecar, its an independent pod sharing pvc with application pod. Here is fluentBit log with Log_Level debug - fluentBit.log – Grandparent

Answer 1

I had the same issue. I was able to get this to work by turning off the Inotify_Watcher setting.

    [INPUT]
        Name                tail
        Tag                 demo.wen.api
        Parser              json
        Path                /var/log/log-*.json
        Mem_Buf_Limit       10MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Inotify_Watcher     false

Fluent Bit Tail Parameters

Lucianolucias

in our setup, we also experienced missing log lines. when we disabled Inotify_Watcher, the problem disappeared. – Theme

Answer 2

Are you continuously writing to the file? In your tail section the parameter ‘read_from_head’ is not used which means that Fluent Bit will only follow the latest data written.

Sawyor

Thanks for replying Anurag. I added the read_from_head option. It now reads all the old logs but does not read new log entries. – Grandparent