Send Cloudwatch logs matching a pattern to SQS queue

Question

I would like to send all Cloudwatch logs where the message of the console.log (appearing in my Cloudwatch logs) matches a certain pattern( for example including the word "postToSlack", or having a certain json field like "slack:true"...)

But I'm stuck at the very beginning of my attempts: I am first trying to implement the most basic task: send ALL cloudwatch logs written when my lambdas are executed (via console.logs placed inside the lambda functions) message to SQS (why? because I first try to make the simplest thing before complexifying with filtering which log to send and which log not to send).

So I created a Cloudwatch Rules > Event > Event Pattern like here below:

{
  "source": [
    "aws.logs"
  ]
}

and as a Target, I selected SQS and then a queue I have created.

But when I trigger for example my lambdas, they do appear in Cloudwatch logs, so I would have expected the log content to be "sent" to the queue but nothing is visible on SQs when I poll/check the content of the queue.

Is there something I am misunderstanding about cloudwatch Rules ?

CONTEXT EXPLANATION

I have lambdas that every hour trigger massively (at my scale:) with like maybe 300 to 500 executions of lambdas in a 1 or 2 minutes period. I want to monitor on Slack all their console.logs (i am logging real error.stack javascript messages as well as purely informative messages like the result of the lambda output "Report Card of the lambda: company=Apple, location=cupertino...").

I could just use a http call to Slack on each lambda but Slack for incoming hooks has a limit of about 1 request per second, after that you get 429 errors if you try to send more than 1 incoming webhook per second... So I thought I'd need to use a queue so that I don't have 300+ lambdas writing to Slack at the same second, but instead controlling the flow from AWS to Slack in a centralized queue called slackQueue.

My idea is to send certain logs (see further down) from Cloudwatchto the SQS slackQueue, and then use this SQS queue as a lambda trigger and sending with this lambda batches of 10 messages (the maximum allowed by AWS; for me 1 message= 1 console.log) concatenated into one big string or array (whatever) to send it to my Slack channel (btw, you can concatenate and send in one call up to 100 slack messages based on Slack limits, so if i could process 100 messages=console.log and concatenate I would but the current batch size limit is 10 for AWS I think ), this way, ensuring I am not sending more than 1 "request" per second to Slack (this request having the content of 10 console.logs).

When I say above "certain logs", it means, I actually I don't want ALL logs to be sent to the queue (because I don't want them on Slack): indeed I don't want the purely "debugging" messages like a console.log("entered function foo"). which are useful during development but have nothing to do on Slack.

As regards some comments: I don't want to use , to my understanding (not expert of AWS) cloudwatch alarms, or metrics filters because they're quite pricy (I'd have those triggered hundreds of times every hour) and don't actually fit my need: I don't want only to read on Slack only when critical problem or a "problem" occurs (like CPU> xxx ...) but really send a regular filtered flow of "almost" all my logs to Slack to read the logs inside Slack instead of inside AWS as Slack is the tool opened all day long, that it's being used for logs/messages coming from other sources than AWS as a centralized place, and that pretty Slack attachment messages formatting is better digested by us. Of course the final lambda (the one sending the messages to slack) would do a bit of formatting to add the italic/bold/etc., and markdown required by slack to have nicely formatted "Slack attachements" but that's not the most complex issue here :)

Thorite

Why do you wish to send the CloudWatch Logs to SQS? The normal mechanism is to specify a Filter Pattern in CloudWatch Logs, which can then be used to trigger a CloudWatch Alarm to alert for particular situations (eg a certain error occurring more then n times per hour). Do you really need to send the content of the CloudWatch Log entry, or are you just trying to trigger some form of notification? – Precede

@JohnRotenstein Thanks for your reply and the help! I think you were right about not giving enough context so I added a detailed explanation on the question. I'm open to any type of system even it's not the one I initially thought best (cloudwatch logs -> filter out useless console.logs -> SQS -> lambda -> Slack) because, to be honest I started AWS a month ago and not expert by any means – Thorite

@Thorite Probably the right tool for the job is SNS. Combined with the Opsidian integration with SLack, you can probably achieve what you need: medium.com/opsidian/… – Steamroller

@SofoGial Opsidian Thanks for your answer. Opsidian looks really cool but the issu is they talk about "Get Intelligent Alarms" on their website so there are 2 main problems with opsidian for me. see next comment – Thorite

1. Setting alarms when a problem occurs as stated in the question, is not my goal. I don't want to only send "problems"/bugs/ with alarms (like CPU > xx) but, as explained in the question, all the logs even they're not an issue (ex: how to filter a log sent by console.log("Report Card of the lambda: company=Apple, location=cupertino... with the system described by your medium post ?); and even if I could "bend" cloudwatch alarms use for sth that is not really an "issue alarm" each time for each log, as they're not crashes/problems, it would be crazy expensive to pipe all logs like this )" – Thorite

2. how does opsidian deal with Slack incoming limits ? I mean I can have 500 new logs such as Report Card of the lambda: company=Apple, location=cupertino. in 2 minutes, I don't think alarms can be throttled/configured in terms of how many per seconds, so they'll break the Slack maximum rate limits for 1 message per second. – Thorite

Seems like cloudwatch logs are not supported for event pattern matching. docs.aws.amazon.com/AmazonCloudWatch/latest/events/… – Civil

Answer 1

@Mathieu, I guess you've misunderstood the CloudWatch Events with CloudWatch logs slightly.

What you need is a real time processing of the log data generated by your lambda functions, filter the logs based on a pattern and then store those filtered logs to your Slack for analysis.

But configuring a CloudWatch Event with SQS is similar like a SQS trigger to Lambda. Here, cloudWatch will trigger (send message to) the SQS queue. The content of the message is not your logs but either the default or custom message that you've created.

Solution #1:

Use Subscription filter to filter out the logs as per requirement and subscribe to AWS Kinesis/AWS Lambda/Amazon Kinesis Data Firehouse. Using the filtered stream (Kinesis), trigger your lambda to push that data to Slack.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

Solution #2:

• Push your cloudWatch logs to S3.
• Create a notification event in S3 on 'ObjectCreated' event and use that to trigger a Lambda function.
• In your Lambda function, write the logic to read the logs from S3 (equivalent to reading a file), filter them and push the filtered logs to Slack.

Pyxis

Thanks, much clearer now. I'll try to go for kinesis i think. thanks a lot for the help – Thorite

I was stuck with this as well. I find the documentation is not really clear here? CloudWatch Events looks like you would be able to stream cloudwatch-logs from external accounts to your own - but thats not possible? – Urbina

@lifeofguenter, yes, exactly. Configuring CloudWatch Event is similar to configuring a trigger point for a job which starts once that event is fired. CloudWatch Logs is whole different service which manages logging on AWS. – Pyxis

Answer 2

@Mathieu, I have the answer where you are stuck.

First whichever lambda function you want to trigger once the pattern gets matched, there you need to add the event like I have added to my lambda function in my serverless yml file as

events:
  - cloudwatchLog:
      logGroup: '/aws/lambda/lambda-function-name'
      filter: '{$.level = 0}'

then

• in your Lambda function, write the logic to read the logs from CloudWatchLogsEvent, you can read it from "cloudWatchLogsEvent.Awslogs.EncodedData".

• Amazon.Lambda.CloudWatchLogsEvents nuget package you need to install for CloudWatchLogsEvent class.

• You need to decode & decompress it. once decompressed you need to Deserialize it to your model.

       byte[] decodedData = Convert.FromBase64String(cloudWatchLogsEvent.Awslogs.EncodedData);
  
       using (var compressedStream = new MemoryStream(decodedData))
       using (var decompressedStream = new MemoryStream())
       using (var gzipStream = new GZipStream(compressedStream, CompressionMode.Decompress))
       {
           gzipStream.CopyTo(decompressedStream);
           var decompressedBytes = decompressedStream.ToArray();
           var decompressedData= Encoding.UTF8.GetString(decompressedBytes);
       }
  
       var logJObject = JObject.Parse(decompressedData);
  

From this JObject you can directly read property like below-

logJObject["logEvents"];

• Like this you need to Filter them and you can use this log for notification(like email you can send using sns topic with email subscription).

I hope this is helpful.

Vacuous