Parsing inner JSON inside FluentD

Question

I have some JSON being emitted from a docker container via the FluentD driver like:

'{"timeMillis":1485917543709,"thread":"main","level":"INFO","loggerName":"com.imageintelligence.ava.api.Boot","message":"{\"dom\":\"DOM\"}","loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":1,"threadPriority":5}'

Notice the message field is string encoded JSON? When this data is captured by fluentD, it ends up looking like this, as expected:

2017-02-01 06:29:15 +0000 docker.6faad650faa6: {"log":"{\"timeMillis\":1485917543709,\"thread\":\"main\",\"level\":\"INFO\",\"loggerName\":\"com.imageintelligence.ava.api.Boot\",\"message\":\"{\\\"dom\\\":\\\"DOM\\\"}\",\"loggerFqcn\":\"org.apache.logging.slf4j.Log4jLogger\",\"threadId\":1,\"threadPriority\":5}\r","com.amazonaws.ecs.cluster":"dombou","container_id":"6faad650faa6012af4f32df79901b42488543a5e6e53517fe3579b01ab2b6862","container_name":"/upbeat_booth","source":"stdout"}`

I use a filter like so, to parse the JSON:

<filter docker.**>
  @type parser
  format json
  key_name log
  reserve_data true
  hash_value_field log
</filter>

and I end up with semi-sanitized JSON:

2017-02-01 06:32:10 +0000 docker.68c794f7f694: {"source":"stdout","log":{"timeMillis":1485917543709,"thread":"main","level":"INFO","loggerName":"com.imageintelligence.ava.api.Boot","message":"{\"dom\":\"DOM\"}","loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":1,"threadPriority":5},"com.amazonaws.ecs.cluster":"dombou","container_id":"68c794f7f6948d4261b9497947834651abbf766e9aa51a76f39d6895b7a9ac18","container_name":"/sad_hamilton"}

The issue is, the message field is still a string escaped JSON field. Any advice on how I can parse that inner JSON field as well? How do I stack filters?

Ossieossietzky

Answer 1

You could try sequential filters:

<filter docker.**>
  @type parser
  key_name log
  format json
  reserve_data true
</filter>

<filter docker.*.embeded_json.**>
  @type parser
  key_name message
  format json
  reserve_data true
</filter>

Andra

You are my savior ! – Obligato

@rhunwicks, it didn't work me. Does the sequential filters still work? – Organometallic

Didnt work for me either (fluentd v1.16) – Decimalize

Answer 2

Please try the following plugin and let me know how it goes:

https://github.com/edsiper/fluent-plugin-docker

Caespitose

My understanding is that this works for the 'log' field, but nothing INSIDE the log field. I need something that will recursively deal with the innards of log. I shall give it a try though – Ossieossietzky

Confirmed: It does not parse the inner json as I want – Ossieossietzky

Didn't work for me either. So even in 2020 we don't have a way to parse inner json? – Organometallic

Answer 3

Define a filter and use json_in_json pluggin for fluentd. After this filter define matcher for this filter to do further process on your log.

Thats helps you to parse nested json. U might also need to add gem install fluent-plugin-json-in-json, if it is not already present. refer - https://github.com/gmr/fluent-plugin-json-in-json

Samuele