Use grok to add the log filename as a field in logstash

Question

I'm using Grok & Logstash to send access logs from Nginx to Elastic search. I'm giving Logstash all my access logs (with a wildcard, works well) and I would like to get the filename (some part of it, to be exact) and use it as a field.

My config is as follows :

input {
  file {
    path => "/var/log/nginx/*.access.log"
    type => "nginx_access"
  }
}

filter {
  if [type] == "nginx_access" {
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}.access.log" }
      add_field => { "app" => "%{app}" }
    }
  }
}
output{
   # whatever
}

But it doesn't seem to work : the app field is added, but has a value of %{app} (not replaced).

I tried different things but to no avail. I may be missing something ... Any ideas ?

Thanks a lot

Hamiltonian

possible duplicate of Logstash: how to add file name as a field? – Beanpole

Answer 1

Ok, found it. grok breaks on match by default. So the first match being good, it skips the second one.

I solved it like that :

filter {
  if [type] == "nginx_access" {
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}.access.log" }
      break_on_match => false
    }
  }
}

Hamiltonian

I'll just note that the field used for messages which were forwarded with logstash-forwarder (aka lumberjack) is not path but file. Everything else works perfectly. – Consultation

Answer 2

I found it more desirable to use 2 grok blocks if there will be unmatching lines in the log files.

filter {
  if [type] == "nginx_access" {
    grok { 
      match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}.access.log" }
    }
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}

Group