Parse Apache2 Error logs with Grok for Logstash

Question

Im trying to parse my apache2 error log and im having a bit of trouble.. It doesnt seem to be matching the filter. Im pretty sure the timestamp piece is wrong, but im not sure, and i cant really find any documentation to figure it out. Also, is there a way to get what is in fields.errmsg to me @message?

Log

[Wed Jun 26 22:13:22 2013] [error] [client 10.10.10.100] PHP Fatal error:  Uncaught exception '\Foo\Bar'

Shipper Config

input {
        file {
                'path' => '/var/log/apache2/*-error.log'
                'type' => 'apache-error'
        }

}

filter {
        grok {
                type => "apache-error"
                pattern => "\[%{HTTPDATE:timestamp}\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}"
        }
}


output {
        stdout {}
        redis {
                'data_type' => 'list'
                'host' => 'logstash.server.net'
                'key' => 'logstash'
        }
}

Idou

Answer 1

Ahoy!

I know I'm a little late to the party, but here it goes!

I created a /etc/logstash/patterns.d/ directory on the system and thew a file named apache-error in it containing:

APACHE_ERROR_TIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}

/etc/logstash/patterns.d/ will be referenced in the logstash configuration as follows:

grok {
  patterns_dir => [ "/etc/logstash/patterns.d" ]
  match => [ "message", "%{APACHE_ERROR_LOG}" ]
}

You can test it out at Grok Debugger, like Adam mentioned in his comment. Seems to work just fine with the sample log entry you send. Has been working pretty solidly me with one of my clients.

The pattern above puts the final message in errormsg field. So I just delete the message field.

This is what I am currently using in my logstash configuration:

filter {
  if [type] == "apache_error_log" {

    grok {
      patterns_dir => [ "/etc/logstash/patterns.d" ]
      match => [ "message", "%{APACHE_ERROR_LOG}" ]
    }

    if !("_grokparsefailure" in [tags]) {

      mutate {
        remove_field => [ "message" ]
        add_field =>  ["timestamp_submitted", "%{@timestamp}"]
      }

      date {
        # Try to pull the timestamp from the 'timestamp' field (parsed above with
        # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
        #                        Sat Feb 08 06:31:09 2014
        match => [ "timestamp", "EEE MMM dd HH:mm:ss yyyy" ]
        remove_field => [ "timestamp" ]
      }

      geoip {
        source => "clientip"
      }
    }
  }
}

Note that I am using a type of apache_error_log instead of apache-error.

Give it s shot. I would love to know if this works for you and others!

Cheers!

Glanti

Thanks! One quick note: I'm using the GELF output plugin to send logs to Graylog2. It requires the message field not be missing, so I had to adjust the pattern accordingly. – Managerial

I'm guessing the log error log format changed since this post, but in case it'll help someone I updated the APACHE_ERROR_LOG pattern to look like this instead: APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}\] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}%{GREEDYDATA:errormsg} – Raddatz

Does not seem to be working for me.. [Sun Oct 11 23:38:09.105800 2015] [mpm_event:notice] [pid 21264:tid 140035264497536] AH00489: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations – Vauban

This one worked for me APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[%{WORD:source}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}%{GREEDYDATA:errormsg} – Vauban

in case anyone else also runs into this, I needed to add an ssl in there: APACHE_ERROR_TIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[ssl:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}\] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}%{GREEDYDATA:errormsg} – Ketubim

Answer 2

Our error logs look a little different:

[Tue Dec 08 12:30:35.997887 2015] [ssl:info] [pid 1178:tid 1072] (70014)End of file found: [client 10.129.24.77:49987] AH01991: SSL input filter read failed.

But I found a predefined pattern that works perfectly:

HTTPD24_ERRORLOG

See this in the Logstash source

Laraelaraine

this is better approach – Ecthyma

worth noting that HTTPD_ERRORLOG will work for both 2.0 and 2.4 formats, so if you're shipping logstash with ansible to a heterogeneous environment (like me) then you should probably use the catch-all version that works for both. – Etiquette

@Robbie Can you please show your filter config, how you have used it? – Onieonion

Answer 3

Didn't work for me:

EEE MMM dd HH:mm:ss yyyy

Worked for me (Apache 2.4):

EEE MMM dd HH:mm:ss.SSSSSS yyyy

Pollypollyanna

Answer 4

Logstash has a built-in apache log parser.

Here is an example...

grok {
   type    => 'company'
   pattern => ["%{COMBINEDAPACHELOG}"]
   add_tag => "apache"
}

As a reference, you can check Logstash's docs

Orjonikidze

Thanks for pointing that out - didn't realize that. You should use the GrokDebugger to check your Grok patterns before putting them in your config file. It's pretty self-explanatory. grokdebug.herokuapp.com – Orjonikidze

nice. That helps. Any idea how to get it so @message is the same as @fields.message tho? – Idou

I'm not sure how that works. I know you can update fields' values by using something like replace => [ "@message", "%{message_remainder}" ], but I don't know how to prepend the @fields to a new message variable. Out of interest, what would you use that for? I can't think of a use case off the top of my head. – Orjonikidze

i dont want the whole log line to be @message, i just want the actual error – Idou

The message itself will always be stored in @message, unless you change this using the replace function. I would recommend continuing on the path you are going (from your snippet in the question). It is a good idea to create a new variable errmsg and then to access the error message just look at @fields.errmsg – Orjonikidze