Regexp JSON filtering in LogQL

Question

I'd like to translate Kibana query like to following to LogQL:

host:("test1-myservice-*") AND level:ERROR 
AND NOT logger_name:"com.example.ExampleClass" 
AND _exists_:stack_trace 
AND NOT stack_trace:(
    "interrupted"
    OR "Read timed out"
    OR "java.lang.InterruptedException"
)

I have tried the following in Grafana Explore but it does not return any records for our JSON log messages:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace =~ ".*InterruptedException.*"

While using != instead of =~ it returns all records:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ ".*InterruptedException.*"

If I'm right the following applies from the documentations to the stack_trace field of the JSON log line:

String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).

Source: Label filter expression

The following seems to work but it seems awkward:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
!~ ".*InterruptedException.*|.*Read timed out.*"
| json

Furthermore, if I'm right, it searches for InterruptedException and Read timed out substrings in the complete JSON string instead of its stack_trace field only.

Is there a more LogQL-ish way to translate the Kibana query above to LogQL? Should the !~ operator work in this case?

Environment: Grafana 7.5.4 / 8.2.3, Loki: 2.4.1

Mccowan

Try LogsQL instead of Loki query language. The query in LogsQL looks almost identical to the original Kibana query. Just remove AND _exists_:stack_trace part, since it isn't needed in LogsQL. – Auberge

Answer 1

This works:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ "(?s).*InterruptedException.*"

Note the (?s) which enables matching new lines for the regex . character. (The stack_trace field of the JSON log message usually contain multiple lines.)

This is also mentioned in the Log stream selector part of the documentation:

Note: The =~ regex operator is fully anchored, meaning regex must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term\n.

Mccowan

Answer 2

Not sure how your log lines look exactly, but I think you don't need to extract the labels out (by using | json

This is a pretty useful article on how to write queries. how-to-create-fast-queries-with-lokis-logql-to-filter-terabytes-of-logs-in-seconds You can also make use of the new Pattern parser instead of the regex if you want to make the query more readable.

So without really knowing how your log lines look, I think this should work well:

{host=~"test1-myservice-.*"}
!= "com.example.ExampleClass" 
!~ ".*InterruptedException.*|.*Read timed out.*"

Based on your needs you can also make use of the Pattern parser I've mentioned before.

Washburn

Thanks for the answer! The first link has a great overview which I've not been completely aware of. Our logs files contain JSON objects in every line. In my example ExampleClass is searched only in the logger_name field and it looks more reliable if we does not hide ExampleClass if it occurs in another, for example, a verbose stack_trace or performance-related JSON field. Any guess for the question related to the !~ operator? I've also updated the question a little bit. – Mccowan

Yes the !~ works after the json if that's what you're asking? – Washburn

Yes, I've asked about that. Thank you for the support, finally I've figured it out (see my answer). – Mccowan

great! glad you have it now :) – Washburn