AWS Elastic Beanstalk: Add custom logs to CloudWatch?
B

6

28

How to add custom logs to CloudWatch? Defaults logs are sent but how to add a custom one?

I already added a file like this: (in .ebextensions)

files:
  "/opt/elasticbeanstalk/tasks/bundlelogs.d/applogs.conf" :
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/app/current/logs/*

  "/opt/elasticbeanstalk/tasks/taillogs.d/cloud-init.conf" :
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/app/current/logs/*

As I did bundlelogs.d and taillogs.d these custom logs are now tailed or retrieved from the console or web, that's nice but they don't persist and are not sent on CloudWatch.

In CloudWatch I have the defaults logs like
/aws/elasticbeanstalk/InstanceName/var/log/eb-activity.log
And I want to have another one like this
/aws/elasticbeanstalk/InstanceName/var/app/current/logs/mycustomlog.log

Berthold answered 18/5, 2017 at 15:49 Comment(0)
H
36

Both bundlelogs.d and taillogs.d are logs retrieved from management console. What you want to do is extend default logs (e.g. eb-activity.log) to CloudWatch Logs. In order to extend the log stream, you need to add another configuration under /etc/awslogs/config/. The configuration should follow the Agent Configuration file Format.

I've successfully extended my logs for my custom ubuntu/nginx/php platform. Here is my extension file FYI. Here is an official sample FYI.

In your case, it could be like

files:
  "/etc/awslogs/config/my_app_log.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/app/current/logs/xxx.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/current/logs/xxx.log"]]}`
      log_stream_name = {instance_id}
      file = /var/app/current/logs/xxx.log*
Hyatt answered 8/6, 2017 at 8:36 Comment(11)
This looks like what I need! Thank you Sebastian!Exhibit
So far I got no luck with this. I compared it with the existing config files already deployed on the EB instance and it looks exactly the same (except for the log paths), so I assume it actually should work. How was your success rate with with @Berthold ? Did it work for you?Evonneevonymus
@TorstenEngelbrecht So I had to "Rebuild" the instance at one moment and it worked after that if I remember well.Exhibit
@Berthold Thanks. This still did not work for me, but I figured out a way to make this work later. I had to attach or extend the policies for the Elastic Beanstalk service role (instance profile) to add more permissions to handle CloudWatch logs. After that they magically appeared in Cloudwatch.Evonneevonymus
Can you explain your complex function for log_group_name? (found some documentation). And please tell me whether the part { "Ref":"AWSEBEnvironmentName" } can somehow be used in the log_stream_name which is what I need to do. What other "Ref" values are possible?Benedikt
I have answered my own question about log_stream_name. I have posted an answer as well :)Benedikt
@TorstenEngelbrecht can you let know the policies you attached to the Elastic Beanstalk service role. I am facing the same issue as yours.Willodeanwilloughby
@Willodeanwilloughby I think this link covers the policy to use: docs.aws.amazon.com/elasticbeanstalk/latest/dg/….Evonneevonymus
Thanks Guys, I got adding CloudWatchLogsFullAccess to my roleGarbers
awslogs troubleshooting doc can be useful: aws.amazon.com/premiumsupport/knowledge-center/… Especially looking at /var/log/awslogs.log. In my case the CreateLogGroup permission was missing, which is not part of docs.aws.amazon.com/elasticbeanstalk/latest/dg/…Supinator
Note that things have changed for the Amazon Linux 2023 platform. AwsLogs has been deprecated in favor of the new Amazon CloudWatch Agent. I've so far been unsuccessful in overwriting the config file for it with platform hooks or eb config files.Mopboard
B
15

Credits where due go to Sebastian Hsu and Abhyudit Jain.

This is the final config file I came up with for .ebextensions for our particular use case. Notes explaining some aspects are below the code block.

files:
  "/etc/awslogs/config/beanstalklogs_custom.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/tomcat8/catalina.out]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Fn::Select" : [ "1", { "Fn::Split" : [ "-", { "Ref":"AWSEBEnvironmentName" } ] } ] }, "var/log/tomcat8/catalina.out"]]}`
      log_stream_name = `{"Fn::Join":["--", [{ "Ref":"AWSEBEnvironmentName" }, "{instance_id}"]]}`
      file = /var/log/tomcat8/catalina.out*

services:
  sysvinit:
    awslogs:
      files:
        - "/etc/awslogs/config/beanstalklogs_custom.conf"

commands:
  rm_beanstalklogs_custom_bak:
    command: "rm beanstalklogs_custom.conf.bak"
    cwd: "/etc/awslogs/config"
    ignoreErrors: true

log_group_name

We have a standard naming scheme for our EB environments which is exactly environmentName-environmentType. I'm using { "Fn::Split" : [ "-", { "Ref":"AWSEBEnvironmentName" } ] } to split that into an array of two strings (name and type).

Then I use { "Fn::Select" : [ "1", <<SPLIT_OUTPUT>> ] } to get just the type string. Your needs would obviously differ, so you may only need the following:

      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat8/catalina.out"]]}`

log_stream_name

I'm using the Fn::Join function to join the EB environment name with the instance ID. Note that the instance ID template is a string that gets echoed exactly as given.

services

The awslogs service is restarted automatically when the custom conf file is deployed.

commands

When the files block overwrites an existing file, it creates a backup file, like beanstalklogs_custom.conf.bak. This block erases that backup file because awslogs service reads both files, potentially causing conflict.

Result

If you log in to an EC2 instance and sudo cat the file, you should see something like this. Note that all the Fn functions have resolved. If you find that an Fn function didn't resolve, check it for syntax errors.

[/var/log/tomcat8/catalina.out]
log_group_name = /aws/elasticbeanstalk/environmentType/var/log/tomcat8/catalina.out
log_stream_name = environmentName-environmentType--{instance_id}
file = /var/log/tomcat8/catalina.out*
Benedikt answered 26/1, 2018 at 22:23 Comment(5)
Sidenote: You can use AWS CLI along with a little utility called cwtail to tail the collected logs.Benedikt
I followed the same steps and reused the code mention in your answer but I couldn't find the file beanstalklogs_custom.conf under /etc/awslogs/config. The .ebextensions folder in my root of the war file. eb activity logs doesnt show any error. is there anything specific to be checked?Willodeanwilloughby
It seems like the prefix "/etc/awslogs/config" is important to getting the log to show up in CloudWatch.Winifield
Thank you for pointing out the backup file, this problem was driving me nuts!!! EB kept using the configuration from the backup instead of the normal one, so it was always using the 'previous build' one.Abbate
Will awslogs know to use the file "beanstalklogs_custom.conf" rather than "beanstalklogs.conf"? Or will it read both?Mannered
H
6

The awslogs agent looks in the configuration file for the log files which it's supposed to send. There are some defaults in it. You need to edit it and specify the files.

You can check and edit the configuration file located at:

/etc/awslogs/awslogs.conf

Make sure to restart the service:

sudo service awslogs restart

You can specify your own files there and create different groups and what not.

Please refer to the following link and you'll be able to get your logs in no time.

Resources:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html

Edit:

As you don't want to edit the files on the instance, you can add the relevant code to the .ebextensions folder in the root of your code. For example, this is my 01_cloudwatch.config :

packages:
  yum:
    awslogs: []

container_commands:
  01_get_awscli_conf_file:
    command: "aws s3 cp s3://project/awscli.conf /etc/awslogs/awscli.conf"
  02_get_awslogs_conf_file:
    command: "aws s3 cp s3://project/awslogs.conf.${NODE_ENV} /etc/awslogs/awslogs.conf"
  03_restart_awslogs:
    command: "sudo service awslogs restart"
  04_start_awslogs_at_system_boot:
    command: "sudo chkconfig awslogs on"

In this config, I am fetching the appropriate config file from a S3 bucket depending on the NODE_ENV. You can do anything you want in your config.

Howze answered 6/6, 2017 at 19:15 Comment(2)
Thanks you Abhyudit, but as it's an Elastic Beanstalk Instance editing the files directly is not something I can do... The instances can be deleted / replaced at anytime!Exhibit
@Berthold You don't have to edit files on the instance. You have to add these commands to .ebextensions and whenever the code is deployed, it will take care of that. See the edit.Howze
T
3

Some great answers already here.

I've detailed in a new Medium blog how this all works and an example .ebextensions file and where to put it.

Below is an excerpt that you might be able to use, the article explains how to determine the right folder/file(s) to stream.

Note that if /var/app/current/logs/* contains many different files this may not work,e.g. if you have

database.log app.log random.log

Then you should consider adding a stream for each, however if you have

app.2021-10-18.log app.2021-10-17.log app.2021-10-16.log

Then you can use /var/app/current/logs/app.*

packages:
  yum:
    awslogs: []

option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 90

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/app/current/logs]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/app/current/logs"]]}`
      log_stream_name = {instance_id}
      file = /var/app/current/logs/*

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd
Thumping answered 18/10, 2021 at 15:26 Comment(0)
G
0

Looking at the AWS docs it's not immediately apparent, but there are a few things you need to do.

(Our environment is an Amazon Linux AMI - Rails App on the Ruby 2.6 Puma Platform).

First, create a Policy in IAM to give your EB generated EC2 instances access to work with CloudWatch log groups and stream to them - we named ours "EB-Cloudwatch-LogStream-Access".

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:log-group:/aws/elasticbeanstalk/*:log-stream:*"
        }
    ]
}

Once you have created this, make sure the policy is attached (in IAM > Roles) to your IAM Instance Profile and Service Role that are associated with your EB environment (check the environment's configuration page: Configuration > Security > IAM instance profile | Service Role).

Then, provide a .config file in your .ebextensions directory such as setup_stream_to_cloudwatch.config or 0x_setup_stream_to_cloudwatch.config. In our project we have made it the last extension .config file to run during our deploys by setting a high number for 0x (eg. 09_setup_stream_to_cloudwatch.config).

Then, provide the following, replacing your_log_file with the appropriate filename, keeping in mind that some log files live in /var/log on an Amazon Linux AMI and some (such as those generated by your application) may live in a path such as /var/app/current/log:

files:
  '/etc/awslogs/config/logs.conf':
    mode: '000600'
    owner: root
    group: root
    content: |
      [/var/app/current/log/your_log_file.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/current/log/your_log_file.log"]]}`
      log_stream_name = {instance_id}
      file = /var/app/current/log/your_log_file.log*
commands:
  "01":
    command: chkconfig awslogs on
  "02":
    command: service awslogs restart # note that this works for Amazon Linux AMI only - other Linux instances likely use `systemd`

Deploy your application, and you should be set!

Garey answered 2/9, 2020 at 19:49 Comment(0)
B
0

In Amazon Linux 2023, awslogs was replaced with amazon-cloudwatch-agent package.

To stream custom logs to CloudWatch you could do the following:

  1. Ensure that your associated EC Service 2 role has necessary permissions to access CloudWatch, create Log Groups, etc.
  2. Ensure that amazon-cloudwatch-agent is installed.
  3. Create custom config that specifies log filename and CloudWatch Log Group name and Stream name.
  4. Append custom config to default config using amazon-cloudwatch-agent-ctl.

Here is an example of .ebextensions/stream_app_logs.config config:

packages:
  yum:
    amazon-cloudwatch-agent: []

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/your-config-name.json" :
    mode: "000755"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/app/current/your-log-filename.log",
                  "log_group_name": "/aws/elasticbeanstalk/YOUR_ENVIRONMENT_NAME/var/app/current/storage/logs/laravel.log",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }

container_commands:
  01_append_config:
    command: "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -c file:/opt/aws/amazon-cloudwatch-agent/etc/your-config-name.json -s"

In my case, I didn't have to restart the service but you can do it like so:

container_commands:
  02_enable_agent:
    command: systemctl enable amazon-cloudwatch-agent.service
  03_restart_agent:
    command: systemctl restart amazon-cloudwatch-agent.service

Troubleshooting tips

  • Ensure the custom config file was created on the instance at /opt/aws/amazon-cloudwatch-agent/etc/your-config-name.json.
  • Ensure the custom config was appended in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml.
  • Check the /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log log file.
  • Ensure amazon-cloudwatch-agent.service is running.
Butterfish answered 20/3 at 13:6 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.