Logstash: how to add file name as a field?

Question

I'm using Logstash + Elasticsearch + Kibana to have an overview of my Tomcat log files.

For each log entry I need to know the name of the file from which it came. I'd like to add it as a field. Is there a way to do it? I've googled a little and I've only found this SO question, but the answer is no longer up-to-date.

So far the only solution I see is to specify separate configuration for each possible file name with different "add_field" like so:

input {
  file {
     type => "catalinalog"
     path => [ "/path/to/my/files/catalina**" ]
     add_field => { "server" => "prod1" }
  }
}

But then I need to reconfigure logstash each time there is a new possible file name. Any better ideas?

Vannoy

Answer 1

Hi I added a grok filter to do just this. I only wanted to have the filename not the path, but you can change this to your needs.

filter {
  grok {
    match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
  }
}

Chimaera

thanks for your answer :) Won't that look for the file name INSIDE the log file content? – Vannoy

No usually you would work with grok on the message field, but I use the path field, which is a special field that is added by logstash automatically. – Chimaera

Thanks a lot :) Your solution works - after adding the percentage sign before "{GREEDYDATA:filename}". Please update your answer and I will accept it as the solution. I just wasn't aware of the existence of the "path" field. – Vannoy

Good one, took out one character to much, fixed it. – Chimaera

@JettroCoenradie: And grok will be added under filter, though what if we want in file input itself? – Niel

There doesn't seem to be a path field as it doesn't work for me in 2.3.3. Does one have to do something to add this field to the event? – Morville

Could be because the field name is source? – Wellfixed

Answer 2

In case you would like to combine the message and file name in one event:

filter {
grok {
    match => { 
        message => "ERROR (?<function>[\S]*)"
        }
}
grok {
    match => { 
        path => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"
        }
}}  

The result in ElasticSearch (focus on 'filename' and 'function' fields):

"_index": "logstash-2016.08.03",
    "_type": "logs",
    "_id": "AVZRyEI49-A6kyBCq6Yt",
    "_score": 1,
    "_source": {
      "message": "27/07/16 12:16:18,321 ERROR blaaaaaaaaa.internal.com",
      "@version": "1",
      "@timestamp": "2016-08-03T19:01:33.083Z",
      "path": "/home/admin/mylog.log",
      "host": "my-virtual-machine",
      "function": "blaaaaaaaaa.internal.com",
      "filename": "mylog"
    }

Hildegaard