fluentd nested json parsing

Question

I have logs like the following:

{
  "log": {
    "header": {
      "key": "value",
      "nested": "{\"key1\":\"value\",\"key2\":\"value\"}",
      "dateTime": "2019-05-08T20:58:06+00:00"
    },
    "body": {
      "path": "/request/path/",
      "method": "POST",
      "ua": "curl/7.54.0",
      "resp": 200
    }
  }
}

I'm trying to aggregate logs using fluentd and I want the entire record to be JSON. The specific problem is the "$.log.header.nested" field, which is a JSON string. How can I parse and replace that string with its contents?

For clarity, I'd like the logs output by fluentd to look like this:

{
  "log": {
    "header": {
      "key": "value",
      "nested": {
          "key1": "value",
          "key2": "value"
      },
      "dateTime": "2019-05-08T20:58:06+00:00"
    },
    "body": {
      "path": "/request/path/",
      "method": "POST",
      "ua": "curl/7.54.0",
      "resp": 200
    }
  }
}

I've found a way to parse the nested field as JSON, but storing to back to the same key it was parsed from isn't clear. It doesn't seem like hash_value_field supports storing to a nested key. Is there some other way to accomplish this?

Bedside

Do you want to keep the original key and value? – Arv

I want to parse the value into a proper object/hash and replace the original value with the parsed value. – Bedside

Answer 1

The following config seems to accomplish what I want. However, I'm not sure if this is the best way. I assume using ruby is far less performant. Any improvements to this are welcome.

<filter logs>
  @type parser
  key_name "$.log.header.nested"
  hash_value_field "parsed_nested"
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

<filter logs>
  @type record_transformer
  enable_ruby true
  <record>
    parsed_nested ${record["log"]["header"]["nested"] = record["parsed_nested"]}
  </record>
  remove_keys parsed_nested
</filter>

Bedside