fluentd: one source for several filters and matches

Question

I have source:

<source>
    @type tail
    tag service
    path /tmp/l.log
    format json
    read_from_head true
</source>

I would like to make several filters on it and match it to several outputs:

<source>
    @type tail
    tag service.pi2
    path /tmp/out.log
    format json
    read_from_head true
</source>

<source>
    @type tail
    tag service.data
    path /tmp/out.log
    format json
    read_from_head true
</source>

<filter service.data>
   # some filtering
</filter>

<filter service.pi2>
   # some filtering
</filter>

<match service.data>
  @type file
  path /tmp/out/data
</match>

<match service.pi2>
  @type file
  path /tmp/out/pi
</match>

So far, to make everything working I have to duplicate source with different tags. Can I make it working from one source definition?

Ligon

Answer 1

You can try using plugins copy and relabel to achieve this. Example configuration looks like this.

//One Source
<source>
    @type tail
    tag service
    path /tmp/l.log
    format json
    read_from_head true
</source>

//Now Copy Source Events to 2 Labels
<match service>
  @type copy
  <store>
    @type relabel
    @label @data
  </store>
  <store>
    @type relabel
    @label @pi2
  </store>
</match>

//@data Label, you can perform desired filter and output file
<label @data>
  <filter service>
    ...
  </filter>
  <match service>
    @type file
    path /tmp/out/data
  </match>
</label>

//@pi2 Label, you can perform desired filter and output file
<label @pi2>
  <filter service>
    ...
  </filter>
  <match service>
    @type file
   path /tmp/out/pi
  </match>
</label>

This Routing examples article has few more ways to do it by re-writing tag etc., but for me I like working with labels and above looks simple.

I have tested above config and it works fine. Let me know your thoughts :).

Aesop

Answer 2

I did it with rewrite_tag_filter.

First I created source by TCP

<source>
  @type tcp
  tag tcp.price-parser
  port 20001
  bind 0.0.0.0
  <parse>
    @type json
  </parse>
</source>

Second step is to match tcp.price-parcer tag and rewrite tag with JSON data

<match tcp.price-parser>
  @type rewrite_tag_filter
  <rule>
    key tag
    pattern /(info|error)/
    tag $1.${tag}
  </rule>
</match>

Its important set rule and match it. If its not matches, fluentd not go further. SO my key is tag. This key is from JSON. For example JSON:

{"tag":"info","message":"My first message"}

And rule pattern regex JSON tag key with /(info|error)/ values. If found info or error we can rewrite fluentd tag. SO tag $1.${tag} is equal info.tcp.price-parser or error.tcp.price-parser.

Now you can match rewrite tag

<match info.tcp.price-parser>
  @type slack
  token xoxb-***
  channel price-parser
  username la
  icon_emoji :ghost:
  message "%s"
  message_keys message
  flush_interval 5s
</match>

Inefficient