CloudWatch multiline log messages from containerized app runnning on ECS/EC2

Question

I have a Web Api (.Net Core 3.0), running on AWS Cluster (ECS), with EC2 instances. In task definition I've configured awslog driver in container to write logs to a certain log group / region.

I'm having problems related to multiline log messages emitted by my Web API (every single line appears splittend in diferrent log messages in CloudWatch). Researching, I've found information that seems to be related to this matter, but I´m not sure about some points.

I've found some information regarding CloudWatch Agent that seems to be related with multiline configuration. Is CloudWatch Agent related with log messages being emitted by my containerized Web API? I mean, my Web API emits log to console and they are automatically thrown to CloudWatch Logs via awslog driver. So, I don't know if Agent in doing anything in this scenario.

The point, is that in CloudWatch Agent documentation, I read something related to a configuration file that seems to be related with the multiline issue:

“multi_line_start_pattern: Specifies the pattern for identifying the start of a log message”

I don't know how the Agent is related to the awslog driver. Regarding to the awslog driver I found these properties related to multiline matter that I could include in my task definition:

“awslogs-multiline-pattern: this option defines a multiline start pattern using a regular expression”

“awslogs-datetime-format: this option defines a multiline start pattern in Python strftime format”

So, both, CloudWatch Agent and awslog driver configuration could resolve the multiline issue? They are related? They are intended for different scopes?

Thanks in advance.

Balfore

Answer 1

Had the same problem. Fixed it by using awslogs-multiline-pattern property on the logdriver configuration.

My application has JSON logs which are either INFO, DEBUG, WARN, ERROR, or CRITICAL so using this property I classify each of these to their own line like so:

LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: some group
              awslogs-stream-prefix: ecs
              awslogs-multiline-pattern: '^(INFO|DEBUG|WARN|ERROR|CRITICAL)'

This is what it looked like prior to using the multiline-pattern:

And this is after:

Hope it helps.

Admittedly

How did you update the task definition with the new log configuration? – Bootless

@Bootless You can do it via CF like this: imgur.com/X6IFnG3 or you can go to ECS console -> Task Definitions and make the change there. – Admittedly

What language is generating the logs? This doesn't work for node console.info calls – Hyoscyamine

Answer 2

@Paras Thakur's answer is only partially correct, and for the wrong reasons. It introduces a new problem whereby the logs aren't going to get sent until the CW agent dumps them (presumably for taking up too much memory), in which case potentially hundreds of logs will have the same timestamp and can be very late arriving. They do get separated into respective messages, but I don't think that's explicitly because if the regex - @Paras Thakur's sample output clearly shows there's no delimiter actually used.

The full solution requires you manually prefix your logs with the correct log-level. Using NodeJS as an example, I think a lot of people who use Lambda will assume that console.info or console.error will create the prefix for you, but that is a customisation to the Lambda runtime that AWS provides.

In ECS you have to add that prefix yourself because you've defined the docker image, not AWS.

In TypeScript you could make a helper like so:

const log = {
    info(...args: Array<any>){
        console.log('INFO', ...args);
    },
    error(...args: Array<any>){
        console.log('ERROR', ...args);
    }
}

log.info('my log', some_object) // "INFO my log { a: 1 }" ...

Here the timestamps are correct (note the presence of the INFO delimiter)

As a bonus, in AWS CDK this would look like:

taskDefinition.addContainer('MyContainer', {
    image: ecs.ContainerImage.fromDockerImageAsset(asset),
    memoryLimitMiB: 512,
    cpu: 256,
    logging: new ecs.AwsLogDriver({
        streamPrefix: 'my-logs',
        multilinePattern: '^(INFO|DEBUG|WARN|ERROR|CRITICAL)'
    })
});

Flamen