How can I view log files in Linux and apply custom filters while viewing?

Question

I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this:

cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less

But it's cumbersome -- every time I want to add another filter, I need to quit less and edit the command line. Some of the filters are relatively complicated and may be multi-line.

I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.

Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.

Changing the code that generates the log to generate less is not an option.

Reeva

I don't have a magic wand for you, but this might be beter on serverfault.com... – Walkout

@Peter -- good suggestion: serverfault.com/questions/117013/… – Reeva

Note that it's considered bad style to start a pipeline with cat FILE | .... Better would be grep args < FILE | ... or just grep args FILE | ... – Venuti

@RSK: Normally I would do something like that, but in reality I have a bunch of grep's piped together and it seemed silly to write: grep -v "OneFilter" < FILE | grep -v "AnotherUglyLongFilter" | grep -v "etc." | less ... I guess just because it buries the filename a bit more. – Reeva

Actually the style of using cat FILE | grep .. | awk ... | sort is more clear. The problem is not style, it is performance, as this way one more process is created and more inter-process communication happens. This is usually no problem for interactive commands, but in scripts the more ugly form should be used. – Martinez

Answer 1

Try the multitail tool - as well as letting you view multile logs at once, I'm pretty sure it lets you apply regex filters interactively.

Deirdra

That looks great... unfortunately not installed. Maybe I'll have to beg the sysadmin. – Reeva

Answer 2

Use &pattern command within less.

From the man page for less

&pattern

          Display  only  lines which match the pattern; lines which do not
          match the pattern are not displayed.  If pattern  is  empty  (if
          you  type  &  immediately  followed  by ENTER), any filtering is
          turned off, and all lines are displayed.  While filtering is  in
          effect,  an  ampersand  is  displayed  at  the  beginning of the
          prompt, as a reminder that some lines in the file may be hidden.

          Certain characters are special as in the / command:

          ^N or !
                 Display only lines which do NOT match the pattern.

          ^R     Don't interpret regular expression  metacharacters;  that
                 is, do a simple textual comparison.

Divulgence

Neat feature. Not helpful for what I was doing 2 years ago but I'll definitely use that one in the future! – Reeva

This is exactly what I was looking for when I found this question. And I think it's exactly what the question owner asked for. This should be the accepted answer! Anyway, many thanks, you saved me a lot of time! – Tragic

@krookedking upgrade to a newer version of less. I had the same problem and upgraded with brew install less, got version 458 which support it. – Edda

Answer 3

Based on ghostdog74's answer and the less manpage, I came up with this:

~/.bashrc:

export LESSOPEN='|~/less-filter.sh %s'
export LESS=-R  # to allow ANSI colors

---

~/less-filter.sh:

#!/bin/sh
case "$1" in
*logfile*.log*) ~/less-filter.sed < $1
  ;;
esac

---

~/less-filter.sed:

/deleteLinesLikeThis/d  # to filter out lines
s/this/that/  # to change text on lines (useful to colorize using ANSI escapes)

---

Then:

• less logfileFooBar.log.1 -- applies the filter applies automatically.
• cat logfileFooBar.log.1 | less -- to see the log without filtering

This is adequate for now but I would still like to be able to edit the filters on the fly.

Reeva

Answer 4

see the man page of less. there are some options you can use to search for words for example. It has line editing mode as well.

Ralli

I need to filter, not search. The INPUT PREPROCESSOR may be helpful, although it's not as dynamic as I wanted. – Reeva

Answer 5

There's an application by Casstor Software Solutions called LogFilter (www.casstor.com) that can edit Windows/Mac/Linux text files and can easily perform file filtering. It supports multiple filters as well as regular expressions. I think it might be what you're looking for.

Harness

That link points to something else entirely now. – Ozone