logstash: multiple logfiles with different pattern

Question

We want to set up a server for logstash for a couple of different project in our company. Now I try to enable them in Kibana. My question is: If I have different patterns of the logfiles, how can I build for them a filter? example: logstash.conf:

input {
  file {
    type => "A"
    path => "/home/logstash/A/*"
    start_position => "beginning"
  }
 file {
    type => "B"
    path => "/home/logstash/B*"
    start_position => "beginning"
  }
}

filter {
  multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
  }
  grok {
     type => A
     match => [ "message", "%{TIMESTAMP_ISO8601:logdate} %{DATA:thread %{LOGLEVEL:level}\s*%{DATA:logger_name}\s*-\s*%{GREEDYDATA:log_text}"]

    add_tag => [ "level_%{level}" ]
  }
  date {
        match => ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"]
  }
  grok {
        type => B
        match => [ any other pattern ... 
 }
}
output {
  elasticsearch { embedded => true }
}

do I have to create for each project (A,B,C,...) an own filter, and what do I have to do, when I have for each project again different pattern of the logfiles?

Preceptor

thanks Ben, I will try it. And what shall I do, when I have different logfile patterns wihtin A and B, when A and B are the projects? – Preceptor

Answer 1

You only need to create a filter for all projects.

For Logstash 1.3.3, You can use if statement to distinct each project grok. For example,

filter {

   multiline {
       pattern => "^%{TIMESTAMP_ISO8601}"
       negate => true
       what => "previous"
   }

   if [type] == "A"  {
      grok {
          match => [ any other pattern ... 
      }
   }
   else if [type] == "B" {
      grok {
          match => [ any other pattern ... 
      }
   }
}

Hope this can help you.

Xeniaxeno

thanks Ben, I will try it. And what shall I do, when I have different logfile patterns wihtin A and B, when A and B are the projects? – Preceptor

One input(file) will have one type. So, if you have different logfile patterns in A, you need to separate the log first. – Xeniaxeno

Thanks Ben.Now for my understanding: When I have two different Webservices (A and B) with different patterns of the logfiles, then it is not possible to display them in one Kibana GUI installation. I would need for every Webservices a own Kibana GUI, right? It is not possible in one GUI with two Dashboards? – Preceptor

I think you have misunderstanding. If you want to use grok to parse your logfiles in Logstash, the log pattern is prefer to same, otherwise your grok pattern will be complicated. Two or more web services log files can save to one elasticsearch(Search Engine). You can have one Kibana GUI with two different dashboard. For example, you can configure one dashboard with type:A and another with type:B – Xeniaxeno