How to exclude namespace from fluent-bit logging

Question

Is there a way to exclude certain namespaces in fluent-bit? I would like to exclude certain namespaces, so that fluent-bit doesn't forward all logs created in those namespaces to ELK.

Is there a way to do it besides adding annotation to each pod in that namespace? I'm aware that you can update all of the pods annotations in a namespace via kubectl.

kubectl annotate pods --namespace=pks-system --all fluentbit.io/exclude='true'

Conservatism

From fluentbit documentation , that is the only way to do it. – Interior

By the way, as the doc describes you can comma separate logs by their namespaces like Exclude_Path /var/log/containers/*_kube-system_*.log,/var/log/containers/*_kubernetes-dashboard_*.log,/var/log/containers/*_yourspecialnamespace_*.log which works for me. – Bibcock

Answer 1

According to official Fluent Bit documentation, for the moment it is actually the unique way of requesting that the log processor skips the logs from certain Pods. I searched through it and found nothing but this fragment.

In addition to that, there is even a feature request raised on their GitHub project so for now we can hope it will be available in a future release.

In documentation there is only example of a separate Pod definition but for sure you should be able to apply it to Pod template in Deployment definition so you don't have to apply it to each Pod separately or to every Pod in certain namespace using the kubectl command you provided.

Scorch

Answer 2

I think the following input plugin configuration can do this:

 [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/*_<myappnamespace>_*.log,/var/log/containers/*_<myappnamespace2>_*.log
        Tag               kube.infra.<namespace_name>.<pod_name>.<container_name>
        Tag_Regex         (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
        Parser            cri
        DB                /var/log/flb_kube_infra.db
        Mem_Buf_Limit     500KB
        Skip_Long_Lines   On
        Refresh_Interval  10

Found it here: https://github.com/fluent/fluent-bit/issues/758

The Exclude_Path property defines the name of the namespace for which logs will be ignored. For multiple logs use comma separated

Succentor

Repeating the full path for every exclusion you make is unnecessary. *search-string-a*,*search-string-b*,... is enough – Epigraphic

Answer 3

You have achieve namespace exclusion with a combination of the three filters kubernetes, nest and grep

[FILTER]
Name                kubernetes
Match               kube.*
Kube_URL            https://kubernetes.default.svc:443
Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix     kube.var.log.containers.
Merge_Log           Off
Merge_Log_Key       log_processed
K8S-Logging.Parser  On
K8S-Logging.Exclude On

[FILTER]
Name                nest
Match               *
Wildcard            pod_name
Operation lift
Nested_under kubernetes
Add_prefix   kubernetes_

[FILTER]
Name                grep
Match               kube.*
Exclude             kubernetes_namespace_name kube-system

Beker

Answer 4

Old question I know. But this works for me and may help others.

[FILTER]
    Name           kubernetes
    Match          kube.*
    # We need the full DNS suffix as Windows only supports resolving names with this suffix
    # See: https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#dns-limitations
    Kube_URL       https://kubernetes.default.svc.cluster.local:443
[FILTER]
    Name           grep
    Match          kube.*
    Exclude        $kubernetes['namespace_name'] kube-system

Deictic

Answer 5

You must read this: https://docs.fluentbit.io/manual/filter/kubernetes#kubernetes-annotations At documentation: "Request to Fluent Bit to exclude or not the logs generated by the Pod. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) have enabled the option K8S-Logging.Exclude."

Woolworth

Thanks. Annotations fit the Kubernetes way of doing things. It would be nice if the filter support not just allowing Pod annotation to exclude, but also Pod annotation to Include...and Namespace annotations too. – Ramble

This should be the preferred answer – Jillane

This doesn't work for me. I have enabled K8S-Logging.Exclude and restarted the daemonset to make sure there's no memory cache, but still logs from the annotated pods are collected. – Helterskelter

@Helterskelter i think if you Enable exclude, you allow people to exclude it via annotations. If you disable it, you do not allow people to exclude it via annotations ( for security purposes ) – Antionetteantioxidant