Create a logStream for each log file in cloudwatchLogs

Question

I use AWS CloudWatch log agent to push my application log to AWS Cloudwatch.

In the cloudwatchLogs config file inside my EC2 instance, I have this entry:

[/scripts/application]
datetime_format = %Y-%m-%d %H:%M:%S
file = /workingdir/customer/logfiles/*.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = /scripts/application

According to this configuration, all log files in workingdir directory are being sent to cloudwatchLogs in the same stream were the name is the instance Id.

My question is, I want for each log file, create a separate logStream, so that the logs reading can be more fast and parseable. In other words, every time I have a new log file, a new logstream is created automatically.

I thought of doing that by a shell script in a cron job but then I'll have to change many other configurations in the architecture, so I'm looking for a way to do it in the config file. In the documentation, they say that :

log_stream_name

Specifies the destination log stream. You can use a literal string or predefined variables ({instance_id}, {hostname}, {ip_address}), or combination of both to define a log stream name. A log stream is created automatically if it doesn't already exist.

The names of the log files can't be 100% predictible, but they always have this structure though:

CustomerName-YYYY-mm-dd.log

Also, another problem is that :

A running agent must be stopped and restarted for configuration changes to take effect.

How can I set the logStream in this case?

Any ideas or suggestions or workarounds are very appreciated.

Multistage

I know you asked this quite some time ago but did you find a solution in the end? I'm experiencing the same limitations in trying to configure the AWS logging agent to be more automated, avoiding the need to configure each log stream separately; we simply have too many log files for this to be feasible, so any input is greatly appreciated. Thanks! – Macron

2024 - the same limitation is still in place – Propitiatory

Answer 1

I know it's been almost two years now, but I wanted to do the exact same thing and couldn't find a way to get it to work. I resorted to the AWS Support, which then confirmed this cannot be done. We're limited to the options offered in the documentation, just like you posted. You can, however, have log groups contain the log file path up to the first dot:

log_group_name – Optional. Specifies what to use as the log group name in CloudWatch Logs. Allowed characters include a-z, A-Z, 0-9, '_' (underscore), '-' (hyphen), '/' (forward slash), and '.' (period).

We recommend that you specify this field to prevent confusion. If you omit this field, the file path up to the final dot is used as the log group name. For example, if the file path is /tmp/TestLogFile.log.2017-07-11-14, the log group name is /tmp/TestLogFile.log.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html

Tropophilous

Answer 2

I was also looking for a similar solution for capturing my app logs, which are written to a log file daily, such that we have a directory with a rolling set of 7 log files (e.g. App-20240829.log, App-20240830.log, etc...). I didn't like that the contents of all 7 log files were written to a single log stream as it reduces readability and is not intuitive for those less familiar with the application.

The only solution I've found for this is to schedule a daily restart of the Cloudwatch Agent at the midnight in concert with the following agent config Logs section, which results in a new log stream being created daily with an associated date appended to the end of the filename.

Below I have included the Logs section of the CloudWatch agent config I am using. You'll notice the inclusion of the {date} variable, which IS NOT listed in official AWS CloudWatch Agent reference documentation (which is quite annoying! I wondering what other variables are available but undocumented...).

"logs": {
    "logs_collected": {
        "files": {
            "collect_list": [
                {
                    "file_path": "<app-path>\*.log",
                    "log_group_name": "<app-name>",
                    "log_stream_name": "<app-name>.{date}",
                    "timezone": "UTC",
                    "timestamp_format": "%Y%m%d.%H%M%S"
                }
            ]
        }
    }
}

Hope this helps, and thanks for the post!

Kook