logstash _grokparsefailure issues

Question

I'm having issues with grok parsing. In ElasticSearch/Kibana the lines I match come up with the tag _grokparsefailure.

Here is my logstash config :

input { 
    file { 
     type => logfile 
     path => ["/var/log/mylog.log"] 
    } 
  } 
filter { 
    if [type] == "logfile" 
    { 
      mutate {
      gsub => ["message","\"","'"]
      }  

    grok 
        { match => { "message" => "L %{DATE} - %{TIME}: " } } 
    } 
} 

output { 
   elasticsearch { host => localhost port => 9300 } 
}

lines/patterns I'm trying to match : L 08/02/2014 - 22:55:49: Log file closed : " finished "

I tried the debugger on http://grokdebug.herokuapp.com/ and it works fine, my pattern matches correctly.

Lines I want to parse might contain double quotes, and I've read there can be issues with the way grok handles and escapes them. So I tried to mutate to replace " with ' to avoid issues but no luck.

Any ideas ? How can I debug this ?

Thanks

Gadolinite

Is there something else in your mylog.log? Because any line that doesn't match will generate a _grokparsefailure. A good strategy for debugging this is to create a test file that has the expected log in it, and use a config file with input { stdin{} } and output {stdout { codec => rubydebug } } and then do logstash -f test_conf < test_file and see what's going on. If you do that and post the input/output, it might be easier to help. As is, your filter is correct for the line given and outputs correctly, although you aren't capturing the grok results... for example{%DATE:date} – Gamboa

Hi, There are other things in the log on which I would expect grokparsefailure yes. Tried using ./logstash -e 'input { stdin{} } output {stdout { codec => rubydebug } }' < mylog.log and got outputs things like : { "message" => "L 08/02/2014 - 22:55:49: Log file closed", "@version" => "1", "@timestamp" => "2014-08-18T19:14:38.033Z", "host" => "lepostack" } Which, seems fine to me :( – Gadolinite

@Gamboa This comment is worth an answer for it answers the question "How can I debug this". I'd give you an upvote, because it helped me. – Roccoroch

Answer 1

Found out the issue, it was around double quotes.

Needed to use simple quote to define the grok filter, and escape double quotes.

match => { 'message' => 'L %{DATE:date} - %{TIME:time}: \"string_between_doublequotes\" '

Gadolinite