Multiple patterns in one log

Question

So I wrote now several patterns for logs which are working. The thing is now, that I have these multiple logs, with multiple patterns, in one single file. How does logstash know what kind of pattern it has to use for which line in the log? ( I am using grok for my filtering ) And if you guys would be super kind, could you give me the link to the docs, because I weren't able to find anything regarding this :/

Humberto

Answer 1

You could use multiple patterns for your grok filter,

grok {
  match => ["fieldname", "pattern1", "pattern2", ..., "patternN"]
}

and they will be applied in order but a) it's not the best option performance-wise and b) you probably want to treat different types of logs differently anyway, so I suggest you use conditionals based on the type or tags of a message:

if [type] == "syslog" {
  grok {
    match => ["message", "your syslog pattern"]
  }
}

Set the type in the input plugin.

The documentation for the currently released version of Logstash is at http://logstash.net/docs/1.4.2/. It probably doesn't address your question specifically but it can be inferred.

Heuser

Answer 2

Write the most specific grok first and use this syntax:

grok {
    match => {
      "message" => [
      #Most specific grok:
        "%{TIMESTAMP_ISO8601:temp_date}%{SPACE}%{LOGLEVEL:log_level}%{UUID:user_id}",
      #Less specific:
        "%{TIMESTAMP_ISO8601:temp_date}%{SPACE}%{GREEDYDATA:log_message}"
     ]
  }
}

Undies

Can the constituent patterns also be defined in this way, or do you have to use regex alternations (|)? – Chatelain

@Chatelain - I'm not sure what you mean by the 'constituent patterns'? – Undies

You can write your own patterns, just like the built in ones like SPACE, LOGLEVEL, UUID, etc. They're expressed as regular expressions, but my issue with that is that having too many alternations in the regex makes it a really long 1-liner. I'm wondering if there is a similar pattern than can be used to define a pattern in terms of an array of regexes, each one tried in turn, just like how the patterns for message are defined here – Chatelain

@Chatelain - Oh now I understand. Well I didn't bump into such feature, but it might be worth to open a new question for this – Undies