AWS Cloudwatch Filter and Pattern Syntax

Question

I'm following the instructions here https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html

but it's not working as i'm expecting it to.

I currently have the following cloudwatch log subscription filter pattern: ? "UNKNOWN_TOPIC_OR_PARTITION" ? " SEVERE " ? " severe " ? " FATAL " ? " fatal " - "closing session"

I would like to match any patter with " fatal " whilst excluding "closing session" from the results.

However, the above filter is matching other log output:

Ordure

Answer 1

You can't with event filter in CloudWatch... but you can with Logs Insights

CloudWatch -> CloudWatch Logs -> Logs Insights

Or

CloudWatch -> CloudWatch Logs -> Log groups -> [your service logs] -> [Button Logs Insights]

Logs Insights

1. Log service (you need to pick what logs of your services will to track
2. In this part you can select the range of time.
3. Here you have your querybox and here you can put querys like an SQL

So in your case you can with this in the query box

fields @timestamp, @message
| sort @timestamp desc
| filter @message like /SEVERE|severe|FATAL|fatal|closing session/ 

Now click on run query and you will see only logs that you want with that filters.

Clute

This doesn't exclude "closing session" – Withal

You can use and @message not like /closing session/ in the filter to exclude the logs with that sentence. – Beadledom

Doesn't answer the question because a log insights expression cannot be used in Filters – Zacynthus

Answer 2

Try this Filter pattern:

[(w1="*UNKNOWN_TOPIC_OR_PARTITION*" || w1="*SEVERE*" || w1="*severe*" || w1="*FATAL*" || w1="*fatal*") && w1!="*closing session*"]

Calculous

This works! But out of curiosity… where is that syntax documented? – Pood

@DanielBang you can find more details here: docs.aws.amazon.com/AmazonCloudWatch/latest/logs/…. Section: Using pattern matching to match terms in space-delimited log events – Calculous

Thanks! I actually looked at that document before coming here but didn’t get to that part. – Pood

I have similar question. #74996339 – Gorky

Answer 3

This bit, in combination with all the ORs, is causing you problems - "closing session". Try removing it a seeing if the rest is matching as expected.

I don't know the syntax to get what you need in a single filter, but to get the same result you can create a separate log filter for each string you want to match. In this case that would be:

• "UNKNOWN_TOPIC_OR_PARTITION" - "closing session"
• " SEVERE " - "closing session"
• " severe " - "closing session"
• " FATAL " - "closing session"
• " fatal " - "closing session"

Now you have 5 different metrics. You can use metric math to sum them up, which will give you the metric you need. See here on how to use metric math:

• Using metric math.
• Alarming on metric math.

Dree

I'm trying to combine all that into a single filter and can't get it to work. So far I' tried: ? "UNKNOWN_TOPIC_OR_PARTITION" - "closing session" ? " SEVERE " - "closing session" ? " severe " - "closing session" ? " FATAL " - "closing session" ? " fatal " - "closing session" – Ordure

My answer is to split it into multiple filters. Not sure how to get it in one filter. – Dree

can't split... you can only have a single cloudwatch log filter subscription per log group which can't be changed.... docs.aws.amazon.com/AmazonCloudWatch/latest/logs/… – Ordure

You are creating metric filters, not subscription filters. Limit is 100. – Dree

Nope, i'm creating subscription cloudwatch filter not metric filter – Ordure

Edit the question then to say so. You have more flexibility in that case, you can do extra processing in the lambda. – Dree

Answer 4

Finally figured out of how to make it work for json log data - mine are fluent bit add-on logs generated from my eks pods , this example works perfectly for me . The xxxxxx in the test data are just me anonymising it.

My use case is to filter by

1. error string in log - "400" or "500"
2. kubernetes.namespace having a string "prod" or "stage" in it

Filter Pattern to generate cloud watch metric filter

{ ($.log = "*400*" || $.log = "*500*" ) && ($.kubernetes.namespace_name = "*prod*" || $.kubernetes.namespace_name = "*stage*")}

Examples for testing

    {"time":"2024-06-20T19:37:54.143939587Z","stream":"stdout","_p":"F","log":"INFO:     xxxxxxx - \"GET /healthz HTTP/1.1\" 200 OK","kubernetes":{"pod_name":"dummy-75c4c7f78d-vv4pk","namespace_name":"dummy-dev","pod_id":"88853af3-7911-4c07-8ab9-d1a90d875242","host":"ip-xxxxxxxxxx.us-east-2.compute.internal","container_name":"dummy","docker_id":"xxxxxxxxxx","container_hash":"xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/dummy@sha256:470eed44a3d65d95def5f8387f2a127f8f29eee94ed14994044093a6ff5332ef","container_image":"xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/dummy:dev-build-1986d44-v1.0.0"}}
{"time":"2024-06-20T19:38:09.143560164Z","stream":"stdout","_p":"F","log":"INFO:     xxxxxxxxxx:36824 - \"GET /healthz HTTP/1.1\" 400 OK","kubernetes":{"pod_name":"dummy-75c4c7f78d-vv4pk","namespace_name":"dummy-prod","pod_id":"88853af3-7911-4c07-8ab9-d1a90d875242","host":"ip-xxxxxxxxxx.us-east-2.compute.internal","container_name":"dummy","docker_id":"xxxxxxxxxx","container_hash":"xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/dummy@sha256:470eed44a3d65d95def5f8387f2a127f8f29eee94ed14994044093a6ff5332ef","container_image":"xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/dummy:dev-build-1986d44-v1.0.0"}}
{"time":"2024-06-20T19:38:24.143866118Z","stream":"stdout","_p":"F","log":"INFO:     xxxxxxxxxx:49414 - \"GET /healthz HTTP/1.1\" 400 OK","kubernetes":{"pod_name":"dummy-75c4c7f78d-vv4pk","namespace_name":"dummy-stage","pod_id":"88853af3-7911-4c07-8ab9-d1a90d875242","host":"ip-xxxxxxxxxx.us-east-2.compute.internal","container_name":"dummy","docker_id":"xxxxxxxxxx","container_hash":"xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/dummy@sha256:470eed44a3d65d95def5f8387f2a127f8f29eee94ed14994044093a6ff5332ef","container_image":"xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/dummy:dev-build-1986d44-v1.0.0"}}

cheers , upvote if this helps someone who came looking for some examples . Putting it here as I landed here while looking for this .

Tip :- ChatGpt and the docs helps :P

Showman