Regular Expressions in Google Cloud Console Logging

Question

How do you search Google App Engine logs in the new Cloud Console using regular expressions?

This blog post suggests you just need to type regex:my.*query to search, but that does not seem to work in the logging console. When I do that, it auto-corrects to the following query text:regex:my.*query.

Mislike

Answer 1

Although, I am late here but you can do it in stackdriver.

=           # equal
!=          # not equal
> < >= <=   # numeric ordering
:           # "has" matches any substring in the log entry field

In case, you want to find all GET response with 500 in textPayload , you need to add in filters:

textPayload:"500"

To search all zone with central1 in it:

resource.labels.zone:"-central1-"

That's it. You can refer this link for more advance filters

Intercut

These are not regular expressions – Ursuline

@Ursuline True, but helps find pattern. Its not an exact answer but its better than nothing. Helped me :) – Intercut

Answer 2

Nowadays Google Cloud Operations Logging has support for regular expressions. This feature was published on 2020-09-17, see https://cloud.google.com/blog/products/management-tools/cloud-logging-gets-regular-expression-support. You regex works in this format:

textPayload=~"my.*query"

You can query regular expression matches with operator =~ and non-matches with operator !~:

=~          # regular expression search for a pattern
!~          # regular expression search not for a pattern

More info on syntax and examples can be found in the Google Cloud Operations Suite Logging query language reference: https://cloud.google.com/logging/docs/view/logging-query-language#regular-expressions

Nival

Answer 3

The Stackdriver Logging product does not currently support regular expressions. It was originally supported a while back (as you saw in the blog post), but we found that it was rarely used and that many of those uses were for simple patterns that had simpler solutions without the performance and other penalties of regexes.

In basic filter mode (the default), text searches automatically are case-insensitive and match substrings of field values, and you can use ".." to represent numerical ranges. In advanced filter mode, the "has" operator accomplishes the same thing through using a : instead of an = in your filter expression, e.g. path.to.field: "value". (See also: Write effective advanced filters)

If these operators don't accomplish your goal, consider filing feedback through the speech bubble button in the top right of the Cloud Console providing details of your use case and what you're ultimately trying to accomplish, and we'll incorporate that feedback as we plan the future direction of the product.

Teddi

Hm. I used regex all the time. Is there any way to easily "AND" two searches together? For example, I want to be able to search for [email protected] and timeout, even though the terms would not be right next to eachother. – Mislike

If your logs are structured, simply searching for both of them does exactly that. If you're just doing free-text searches, you'd need to use advanced mode, since basic mode ORs like-typed searches together, while in advanced mode everything is ANDed by default (and you can really do anything you want there anyway). – Teddi

Okay thanks.. The default ORing in the simple mode really through me off. I've never seen a search interface default to ORing terms together. Would be nice if you could add search terms without going into advanced mode. – Mislike

FWIW, it's a little more nuanced than that in basic mode. Different kinds of constraints are ANDed, like kinds of constraints are ORed: if you wrote path:/foo path:/bar, you probably don't want that to return nothing because no single entry has both of those paths simultaneously. – Teddi

That's an incredibly confusing search interface right out of the box. Has the team considered moving to just using AND? Pretty much all search engines (e.g., google) work that way. – Mislike

Google web search actually doesn't work that way. =) In any case, we inherited this from the old GAE viewer and customers have dependences on this behavior, and it generally works better for basic use cases, even if it is a gotcha for advanced use cases. The new advanced filter interface does indeed have default AND across all clauses, and we expect more people to use it by default over time as we make some changes we have in the queue for it. – Teddi

Yes, to be pedantic, I'm sure Google's search algorithm is much more complex than ANDing terms together. That said, it is generally the case that more search terms leads to narrower results. I'm not sure I've seen a search bar that does the opposite (besides Cloud Console). – Mislike

Yeah, makes no sense to use OR IMHO. Same feedback here, switch to AND. – Key