logstash grok filter for logs with arbitrary attribute-value pairs

Question

(This is related to my other question logstash grok filter for custom logs )

I have a logfile whose lines look something like:

14:46:16.603 [http-nio-8080-exec-4] INFO  METERING - msg=93e6dd5e-c009-46b3-b9eb-f753ee3b889a CREATE_JOB job=a820018e-7ad7-481a-97b0-bd705c3280ad data=71b1652e-16c8-4b33-9a57-f5fcb3d5de92
14:46:17.378 [http-nio-8080-exec-3] INFO  METERING - msg=c1ddb068-e6a2-450a-9f8b-7cbc1dbc222a SET_STATUS job=a820018e-7ad7-481a-97b0-bd705c3280ad status=ACTIVE final=false

I built a pattern that matched the first line:

%{TIME:timestamp} %{NOTSPACE:http} %{WORD:loglevel}%{SPACE}%{WORD:logtype} - msg=%{NOTSPACE:msg}%{SPACE}%{WORD:action}%{SPACE}job=%{NOTSPACE:job}%{SPACE}data=%{NOTSPACE:data}

but obviously that only works for lines that have the data= at the end, versus the status= and final= at the end of the second line, or other attribute-value pairs on other lines? How can I set up a pattern that says that after a certain point there will be an arbitrary of foo=bar pairs that I want to recognize and output as attribute/value pairs in the output?

Bouchard

Answer 1

You can change your grok pattern like this to have all the key value pairs in one field (kvpairs):

%{TIME:timestamp} %{NOTSPACE:http} %{WORD:loglevel}%{SPACE}%{WORD:logtype} - %{GREEDYDATA:kvpairs}

Afterwards you can use the kv filter to parse the key value pairs.

kv {
    source => "kvpairs"
    remove_field => [ "kvpairs" ] # Delete the field afterwards
}

Unfortunately, you have some simple values inside your kv pairs (e.g. CREATE_JOB). You could parse them with grok and use one kv filter for the values before and another kv filter for the values after those simple values.

Dubai

This looks like exactly what I need -- one question, can I suppress the output of the entire kvpairs value in my output? – Bouchard