Logstash close file descriptors?

Question

BACKGROUND:

We have rsyslog creating log files directories like: /var/log/rsyslog/SERVER-NAME/LOG-DATE/LOG-FILE-NAME So multiple servers are spilling out their logs of different dates to a central location.

Now to read these logs and store them in elasticsearch for analysing I have my logstash config file something like this:

file{
   path => /var/log/rsyslog/**/*.log
}

ISSUE :

Now as number of log files in the directory increase, logstash opens file descriptors (FD) for new files and will not release FDs for already read log files. Since log files are generated per date, once it is read, it is of no use after that since it will not be updated after that date.

I have increased the file openings limit to 65K in /etc/security/limits.conf

Can we make logstash close the handle after some time so that number of file handles opened do not increase too much ??

Delores

What Logstash version is this? Also, can you post the complete config file? – Lorie

Asking because of this: github.com/elastic/logstash/issues/1604. Do you have the same symptoms? Exceptions in logs after some time? If you run sudo lsof | grep java | wc -l do you see the descriptors steadily increasing over time?\ – Lorie

Answer 1

I think you may have hit this bug: http://github.com/elastic/logstash/issues/1604. Do you have the same symptoms? Exceptions in logs after some time? If you run sudo lsof | grep java | wc -l do you see the descriptors steadily increasing over time? (some of them might close, but some will stay open and their number will increase)

Lorie

Answer 2

I've been tracking this issue for some time, and I don't know that it's properly solved.

We were in a similar boat, perhaps bigger: Logstash couldn't open handles for hundreds of thousands of log files on a box, even though very few of them written to actively. LOGSTASH-271 captured this issue, and there were some attempts to patch Logstash, including PR #1260.

It seems a fix may have made it's way into Logstash 1.5 with PR #1545, but I've never tested this personally. We ended up forking the underlying library Logstash uses to implement the file input, called FileWatch, into FFileWatch, which adds an "eviction mechanism".

The basic idea behind this approach is to only keep files open while they're being written. Normally, Logstash will open a handle on the file and keep it open forever, but FFileWatch adds an option to close the handle if the file has not changed recently (eviction_interval). I then created a custom build of Logstash using the forked gem.

Obviously this is less than ideal, but it worked for us. Eventually we dropped Logstash entirely for picking up log files, although we still use it further down the log processing pipeline. We implemented our own lightweight log shipper (Franz), which does not suffer from this issue.

Euryale