How to filter logs in Grafana-Loki?

Question

I use the last versions of:

• Prometheus
• prometheus-cpp library
• Grafana
• Loki
• Promtail

in Windows 10.

So I just start bin files of these applications. And I get my logfile.log in Grafana panel.

There are lines marked [INFO] in the log file.
There may also be an [ERROR] mark. I want to be able to filter rows in a panel. For example, how can I display only lines marked with [ERROR] in the panel?

Brigid

Answer 1

One can use line filters to accomplish that:

|= : Log line contains string.
!= : Log line does not contain string.
|~ : Log line matches regular expression.
!~ : Log line does not match regular expression.

For instance:

A given log line from target cluster and app contains the string ERROR:

{cluster="my-dev-cluster", app="my-app"} |= "ERROR"

A given log line from target cluster and app contains the string ERROR but does not contain the string "TIMEOUT":

{cluster="my-dev-cluster", app="my-app"} |= "ERROR" != "TIMEOUT"

See more in this cheat-sheet.

Khz

Answer 2

Loki indexes logs by their labels. When you query Loki you must first specify a stream filter in the format {<label><operator><value>}. I'm guessing you're already aware of this since you've got your logs being displayed there, probably with something like {app="vocoder"}. You can then filter the stream of logs by plain text or regular expression with something like {app="vocoder"} |= "[ERROR]" (see the Log queries documentation). If you configure your Promtail scrapers to extract additional labels from the log messages (see Labels documentation and Scraping documentation), then you can also write stream filter expressions based on this, or filter based on this labels as part of your filter pipeline: {app="vocoder"} | level = "error".

Fontenot

+1 to what @Paul said. You can use LogQL in Grafana's Explore view grafana.com/docs/grafana/latest/explore and play around with filtering your logs. – Nessim