conditional matching with grok for logstash

Question

I have php log of this format

[Day Mon DD HH:MM:SS YYYY] [Log-Type] [client <ipv4 ip address>] <some php error type>: <other msg with /path/of/a/php/script/file.php and something else>
[Day Mon DD HH:MM:SS YYYY] [Log-Type] [client <ipv4 ip address>] <some php error type>: <other msg without any file name in it>
[Day Mon DD HH:MM:SS YYYY] [Log-Type] [client <ipv4 ip address>] <some msg with out semicolon in it but /path/of/a/file inside the message>

This I am trying to send to Graylog2 after processing through logstash. Using this post here, I was able to start. now I would like to get some additional fields, so that my final version would look something like this.

{
       "message" => "<The entire error message goes here>",
      "@version" => "1",
    "@timestamp" => "converted timestamp from Day Mon DD HH:MM:SS YYYY",
          "host" => "<ipv4 ip address>",
       "logtime" => "Day Mon DD HH:MM:SS YYYY",
      "loglevel" => "Log-Type",
      "clientip" => "<ipv4 ip address>",
      "php_error_type" => "<some php error type>"
      "file_name_from_the_log" => "/path/of/a/file || /path/of/a/php/script/file.php"
      "errormsg" => "<the error message after first colon (:) found>"
}

I have the expression for individual line, or atleast I think these should be able to parse, using grokdebugger. something like this:

%{DATA:php_error_type}: %{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}
%{DATA:php_error_type}: %{GREEDYDATA:errormsg}
%{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}

But somehow I am finding it very difficult to make it work for the entire log file.

Any suggestion please? Also, not sure if there would be any other type of error messages coming in the log file. but the intention is to get the same format for all. Any suggestions how to tackle these logs to get the above mentioned format?

Jalapa

What does "very difficult" mean? Does one of your patterns not work in the debugger, or ?? – Retractile

I am not sure if I have asked the question correctly. I actually want to use all of these conditions in the configuration file, so that all lines are parsed and i get the same output. As of now I am not sure, how to add them, where to add them. – Jalapa

Answer 1

The grok filter can be configured with multiple patterns:

grok {
  match => [
    "message", "%{DATA:php_error_type}: %{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}",
    "message", "%{DATA:php_error_type}: %{GREEDYDATA:errormsg}",
    "message", "%{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}"
  ]
}

(Instead of a single filter with multiple patterns you could have multiple grok filters, but then you'd probably want to disable the _grokparsefailure tagging with tag_on_failure => [].)

Kapellmeister

i came here to write the same thing. Actually, i did see this post initially, but didn't understand. Today i was able to. so now i got that to be working. Thanks none the less. a super happy logstash + graylog2 user :). – Jalapa

Hi @Magnus For me i am facing invalid configuration – Schecter

@feelgoodandprogramming Correct syntax is grok { match => { "message" => ["%{DATA:php_error_type}: %{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}", "%{DATA:php_error_type}: %{GREEDYDATA:errormsg}", "%{DATA:message_part1}%{URIPATHPARAM:file_name}%{GREEDYDATA:errormsg}",] } } – Moral

@SaurabhSaxena Both forms are valid. However, there was a trailing comma after the last grok expression in my post which would cause Logstash to reject it. I've removed it. – Arroba

Answer 2

If you have some part of your log line missing sometime you can use the following syntax :

(?:%{PATTERN1}|%{PATTERN2})

or

(?:%{PATTERN1}|)

To allow PATTERN1 OR ''. (empty)

Using this, you can have have only one pattern to manage :

grok {
   match => [
      "message", "(?:%{DATA:php_error_type}: |)(?:%{DATA:message_part1}:)(?:%{URIPATHPARAM:file_name}|)%{GREEDYDATA:errormsg}",
   ]
}

If you have problems, maybe replace %{DATA} by a more restrictive pattern.

You can also use this syntax (more regex like)

(?:%{PATTERN1})?

To debug a complex grok pattern, I recommend :

• https://grokconstructor.appspot.com/do/match (multiline option + multiple input lines at same time + others options)
• https://grokdebug.herokuapp.com/ (simpler to use)

Therewithal