How can I drop an empty line in logstash

Question

In my logstash logs I have sometimes empty lines or lines with only spaces.

To drop the empty line I created a dropemptyline filter file

# drop empty lines
filter {
    if [message] =~ /^\s*$/ {
        drop { }
    }
}

But the empty line filter is not working as expected, mainly because this particular filter is inside a chain other filters where there are filter coming afterwards afterwards.

00_input.conf
05_syslogfilter.conf
06_dropemptylines.conf
07_classifier.conf

So I think my particular filter would work if it was the only one but its not.

2015-02-11 15:02:12.347  WARN 1 --- [tp1812226644-23] o.eclipse.jetty.servlet.ServletHandler   : 

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]

My question is how can I drop out of all filters and go directly to output?

Narvaez

Based on your sample data, it looks like a multiline message that has been combined. If so, the extra newline is inside your string, so anchoring the regexp (with ^ and $) won't work. – Voluntary

the multiline filter is after droplines. – Narvaez

Passing your input through a simple config with just the drop{} works as expected. Do you have more details on the issue? – Voluntary

Answer 1

you can just ignore empty lines entirely using a grok filter,

%{GREEDYDATA:1st}(\n{1,})%{GREEDYDATA:2nd}

it will generate,

{
  "1st": [
    [
      "2015-02-11 15:02:12.347  WARN 1 --- [tp1812226644-23] o.eclipse.jetty.servlet.ServletHandler   : "
    ]
  ],
  "2nd": [
    [
      "org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]"
    ]
  ]
}

or more elegant way,

(?m)%{GREEDYDATA:log}

Output:

{
  "log": [
    [
      "2015-02-11 15:02:12.347  WARN 1 --- [tp1812226644-23] o.eclipse.jetty.servlet.ServletHandler   : \n\n\n\norg.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongo:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.UnknownHostException: mongo: unknown error}}]"
    ]
  ]
}

Verruca