Multiple Grok Filters not storing first filter match record

Question

I am using Logstash to parse postfix logs. I am mainly focused to get bounced email logs from postfix logs, and store it in database.

In order to get logs, first I need to find ID generated by postfix corresponding to my message-id, and using that Id, I need to find status of an email. For following configuation, I am able to get the logs.

grok {
       patterns_dir => "patterns"
       match => [
            "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}",
            "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}"
        ]
        named_captures_only => true
    }

I am using following if condition to store logs that match the patterns:

if "_grokparsefailure" not in [tags] {
   #database call
}

As you have seen, I am using two patterns to find corresponding two different logs from one log file.

Now, I want to differentiate both pattern based on tags. So I have modified my configuration as follows:

  grok {
       patterns_dir => "patterns"
       match => [
            "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}"
        ]
        add_tag => ["BOUNCED"]
        remove_tag => ["_grokparsefailure"]
        named_captures_only => true
    }

    grok {
       patterns_dir => "patterns"
       match => [
            "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}"            
        ]
        add_tag => ["INTIALIZATION"]
        remove_tag => ["_grokparsefailure"]
        named_captures_only => true
    }

Now, it only store %{POSTFIXCLEANUP} pattern logs. If I reverse the order, it only store %{POSTFIXBOUNCE} pattern.

so, after removing that if condition, I found that message being parsed from first filter have "_grokparsefailure" tag and first filter tag, and because of that it is not storing that record.

Can anybody tell me what need to be done to rectify this? Am I am making any mistake?

Tamworth

pretty sure the remove_tag only gets done if it matches the event, so that's likely your issue – Dibb

So, what do i need to do in order to achieve desired results? – Tamworth

Answer 1

You need to protect the 2nd grok block -- ie don't execute it if the first one succeeds.

if ("BOUNCED" not in [tags]) {
  grok {
    patterns_dir => "patterns"
    match => [
        "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}"            
    ]
    add_tag => ["INTIALIZATION"]
    remove_tag => ["_grokparsefailure"]
    named_captures_only => true
  }
}

Dibb

if statement should be if( "BOUNCED" not in [tags]){} – Leptospirosis