have a grok filter create nested fields as a result

Question

I have a drupal watchdog syslog file that I want to parse into essentially two nested fields, the syslog part and the message part so that I get this result

syslogpart: {
  timestamp: "",
  host: "",
  ...
},
messagepart:{
  parsedfield1: "",
  parsedfield2: "",
  ...
}

I tried making a custom pattern that looks like this:

DRUPALSYSLOG (%{SYSLOGTIMESTAMP:date} %{SYSLOGHOST:logsource} %{WORD:program}: %{URL:domain}\|%{EPOCH:epoch}\|%{WORD:instigator}\|%{IP:ip}\|%{URL:referrer}\|%{URL:request}\|(?<user_id>\d+)\|\|)

and then run match => ['message', '%{DRUPALSYSLOG:drupal}'}

but I don't get a nested response, I get a textblock drupal: "ALL THE MATCHING FIELDS IN ONE STRING" and then all the matches separately as well but not nested under drupal but rather on the same level.

Prefer

Answer 1

Yes, this is expected. I don't think there's a way to produce nested fields with grok. I suspect you'll have to use the mutate filter to move them into place.

mutate {
    rename => {
      "date" => "[drupal][date]"
      "instigator" => "[drupal][instigator]"
      ...
    }
  }

If you have a lot of fields it might be more convenient to use a ruby filter. This is especially true if you prefix Drupal fields with e.g. "drupal." – then you'd write a filter to move all fields with that prefix into a subfield with the same name.

Larkin

The bracket syntax is supported directly in the pattern definition. No need for mutate/rename! See Julien's answer below. – Tedford

Answer 2

Actually, you can do something like that in your pattern config

%{WORD:[drupal][program]}

It will create the json object like

drupal:{
  program: "..."
}

Coniine

Yes, that does work except if you want to create subfields of @metadata (or, presumably, other fields with unusual characters in the name). – Shockley

Can confirm, still works with logstash-7.11.0. Example: APPLICATION (?:\[%{WORD:[application][context]}\])?\[%{WORD:[application][messageid]}\]\[%{WORD:[application][logcategory]}\]\[%{WORD:[application][logpriority]}\] – Yerxa