Logstash replace @timestamp with syslog date

Question

I'm a bit confused. I'm trying to pull out the syslog date (backfilling the logstash) and replace the @timestamp with it. I've tried almost everything.

This is my filter

filter {
   if [type] == "syslog" {
   grok {
     match => {
"message" => ["%{SYSLOGTIMESTAMP:DATETIME} %{WORD:SERVER} (?<BINARY>(.*?)(php\-cgi|php))\: %{DATA:PHP_ERROR_TYPE}\:\s\s(?<PHP_ERROR_DESC>(.*?)(e\s\d))""]
  }
}

date {
  match => { "DATETIME" => [ "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] }
  target => "@timestamp"
  add_tag => [ "tmatch" ]
}

if !("_grokparsefailure" in [tags]) {
  mutate {
    replace => [ "@source_host", "%{SERVER}" ]
  }
}
mutate {
  remove_field => [ "SERVER" ]
}
}
}

sample output:

{
    "message" => "Sep 10 00:00:00 xxxxxxx",
    "@timestamp" => "2013-12-05T13:29:35.169Z",
      "@version" => "1",
          "type" => "xxxx",
          "host" => "127.0.0.1:xxx",
      "DATETIME" => "Sep 10 00:00:00",
        "BINARY" => "xxxx",
"PHP_ERROR_TYPE" => "xxxx",
"PHP_ERROR_DESC" => "xxxxx",
          "tags" => [
    [0] "tmatch"
],
  "@source_host" => "xxx"
}

tmatch is in the tags so I assume that the date filter works, but why do I still have:

@timestamp => "2013-12-05T13:29:35.169Z"

?

Thanks for help (my logstash is logstash-1.2.2-flatjar.jar)

Intumesce

Answer 1

Let's take a look at your date filter:

date {
  match => { "DATETIME" => [ "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] }
  target => "@timestamp"
  add_tag => [ "tmatch" ]
}

In particular, the match parameter:

match => { "DATETIME" => [ "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] }

Match expects an array. I'm not sure what you're passing, exactly, but it's definitely not an array. I tried running this with -v, and I'm surprised to see it doesn't complain.

You probably mean something closer to this:

match => ["DATETIME", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]

Note the first element of the array is the target field; additional elements are pattern(s) to match against.

Past that, you really only need to pass the one format you expect, but it looks like that's included among the three you're sending.

Reynoso

hey, thanks for response! I've read that the construction of the match has changed recently. But I've tried your solution and it still doesn't work. I have reasonable assumpitions that the Joda time library is an issue. Bug is reported on Logstash JIRA: logstash.jira.com/browse/LOGSTASH-1686 – Intumesce

Good to know. I'll keep an eye out for that in the future. – Reynoso

Answer 2

If you want the timestamp showed as your timezone format, instead of UTC time, you can do like this

ruby {
    code => "event['@timestamp'] = event['@timestamp'].local('-08:00')"
}

Before:@timestamp => "2013-12-05T13:29:35.169Z"

After :@timestamp => "2013-12-05T05:29:35.169-08:00"

Updated: The local method can't work in version 1.4.2. So, change another API

ruby {
    code => "event['@timestamp'] = event['@timestamp'].getlocal"
}

Deluna

This doesn't really work. It gives the following error "exception"=>#<NoMethodError: undefined method `local' for nil:NilClass>. Have you tried this ? – Ky

I'm using log stash 1.4.2. I need to be able to convert the date to UTC format when indexing in ES. Any ideas how can this be achieved ? – Ky

Thanks. This works! But it only works with metadata fields like @timestamp. When I try to use it with a field which has the similar timestamp it does not. I try doing this, code => "event['syslog_timestamp'] = event['syslog_timestamp'].getlocal('-08:00')" But it fails with NoMethodError..do you have any idea as to why ? – Ky

Please make sure your field is in a timestamp, type, not string. When u parse your logs, maybe the field format is string, so your need to cast it to timestamp type – Deluna