How to stream custom logs to cloudwatch from AL2023 instances

Question

We use amazon-elastic-beanstalk to deploy part of our stack. We have just migrated our base platform from PHP 7.4 which uses Amazon Linux 2, to PHP 8.2 which uses Amazon Linux 2023.

Default system logs are being streamed properly (/var/log/nginx/access.log, /var/log/eb-hooks.log, etc.), but custom log streaming to cloudwatch is not working.

I followed this guide https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html in order to enable log streaming within amazon-linux-2

(The guide has not been updated to amazon-linux-2023)

In AL2 we declare the logs we want to be sent creating an ebextension that added the files within /etc/awslogs/config/logs.conf like described in this example from awsdocs

Also I had to enable Cloudwatch log streaming by creating another ebextension: .ebextension/95-logs-streamtocloudwatch.config

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

And finally I created a policy and add it to the role defined in the EC2 instance profile like documented in the official guide: Using Elastic Beanstalk with Amazon CloudWatch Logs

{
 "Version": "2012-10-17",
 "Statement": [
 {
   "Effect": "Allow",
   "Action": [
     "logs:CreateLogGroup",
     "logs:CreateLogStream",
     "logs:PutLogEvents",
     "logs:DescribeLogStreams"
   ],
   "Resource": [
   "*"
   ]
 }
 ]
}

from: official source

That configuration worked fine within amazon-linux-2 instances. But is not working within Amazon Linux 2023 instances

Beachlamar

Answer 1

I have found the solution thanks to this existing answer

What I have done is to edit the .ebextension file in order to set the configuration for the cloudwatch agent under /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ folder.

This is my resulting .ebextension file

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

packages:
  yum:
    amazon-cloudwatch-agent: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`
  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/custom_logs.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/php-fpm/mycustomlog.log",
                  "log_group_name": "/aws/elasticbeanstalk/erp-prod-php8/var/log/php-fpm/mycustomlog.log",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }

commands:
  "01":
    command: systemctl enable amazon-cloudwatch-agent.service
  "02":
    command: systemctl restart amazon-cloudwatch-agent

Beachlamar

Has this changed since the answer was posted? Whatever I write to amazon-cloudwatch-agent.d gets wiped and my understanding is awslogs doesn't exist anymore and is not used by the new agent. I ssh'd in and the awslogs directory is not there. What are those awslog files doing in this answer? – Boris

That solution It's still working on my evironments. What do you mean by "gets wiped"? in elastic beanstalk context, whatever you write into amazon-cloudwatch-agent.d will be removed after every deploy – Beachlamar

I added a file to the agent.d folder in the Prebuild platform hook. In the script I made sure it wrote properly. Then after deploy finished, the file was gone and it did not add my log file to CloudWatch. I also tried an .ebextension like you did and it also did not work for me. But I did not try a restart of the agent. Maybe that is what I'm missing. I'll try it again – Boris

Answer 2

The config mentioned above did not work for me. I created an cloudwatch config file using the commands mentioned in the other link Paul added as a reference and it worked! (attached logs). Also the cloudwatch agent is restarted as per the commands (see logs), but for some reason the logs don't stream until I manually ssh into the EC2 instance and run those 3 commands, again. Did anyone face this issue? If yes, appreciate if you can share how you solved it. Here's the config.

packages:
  yum:
    amazon-cloudwatch-agent: []
option_settings:
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: StreamLogs
    value: true
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: DeleteOnTerminate  
    value: false
  - namespace: aws:elasticbeanstalk:cloudwatch:logs
    option_name: RetentionInDays
    value: 90
files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`
  "/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/app/current/logs/spring.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/current/logs/spring.log"]]}`",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }
commands:
  "01":
    command: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json -s
  "02":
    command: sudo systemctl enable amazon-cloudwatch-agent.service
  "03":
    command: sudo systemctl restart amazon-cloudwatch-agent

and here's the result from the EB logs

2024-02-28 22:19:04,450 P38264 [INFO] ============================================================
2024-02-28 22:19:04,450 P38264 [INFO] yum list installed amazon-cloudwatch-agent
2024-02-28 22:19:04,763 P38264 [INFO] -----------------------Command Output-----------------------
2024-02-28 22:19:04,763 P38264 [INFO]   Installed Packages
2024-02-28 22:19:04,763 P38264 [INFO]   amazon-cloudwatch-agent.x86_64        1.300032.3-1.amzn2023         @amazonlinux
2024-02-28 22:19:04,763 P38264 [INFO] ------------------------------------------------------------
2024-02-28 22:19:04,763 P38264 [INFO] Completed successfully.
2024-02-28 22:19:04,766 P38264 [INFO] ============================================================
2024-02-28 22:19:04,766 P38264 [INFO] Command 01
2024-02-28 22:19:05,898 P38264 [INFO] -----------------------Command Output-----------------------
2024-02-28 22:19:05,898 P38264 [INFO]   ****** processing amazon-cloudwatch-agent ******
2024-02-28 22:19:05,898 P38264 [INFO]   I! Trying to detect region from ec2 D! [EC2] Found active network interface I! imds retry client will retry 1 timesSuccessfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_custom_logs.json.tmp
2024-02-28 22:19:05,898 P38264 [INFO]   Start configuration validation...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_beanstalk.json ...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_custom_logs.json.tmp ...
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 I! Valid Json input schema.
2024-02-28 22:19:05,898 P38264 [INFO]   2024/02/28 22:19:05 Configuration validation first phase succeeded
2024-02-28 22:19:05,898 P38264 [INFO]   /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2024-02-28 22:19:05,898 P38264 [INFO]   I! Detecting run_as_user...
2024-02-28 22:19:05,899 P38264 [INFO]   I! Trying to detect region from ec2
2024-02-28 22:19:05,899 P38264 [INFO]   D! [EC2] Found active network interface
2024-02-28 22:19:05,899 P38264 [INFO]   I! imds retry client will retry 1 times
2024-02-28 22:19:05,899 P38264 [INFO]   /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2024-02-28 22:19:05,899 P38264 [INFO]   Configuration validation second phase succeeded
2024-02-28 22:19:05,899 P38264 [INFO]   Configuration validation succeeded
2024-02-28 22:19:05,899 P38264 [INFO] ------------------------------------------------------------
2024-02-28 22:19:05,899 P38264 [INFO] Completed successfully.
2024-02-28 22:19:05,899 P38264 [INFO] ============================================================
2024-02-28 22:19:05,899 P38264 [INFO] Command 02
2024-02-28 22:19:06,292 P38264 [INFO] Completed successfully.
2024-02-28 22:19:06,293 P38264 [INFO] ============================================================
2024-02-28 22:19:06,293 P38264 [INFO] Command 03
2024-02-28 22:19:06,431 P38264 [INFO] Completed successfully.
2024-02-28 22:19:13,835 P38770 [INFO] ************************************************************

Edit:

Alright for those who are facing same problem, I figured it out. I had to use container_commands instead of commands. container_commands are executed after the app is deployed, at least that's how it is in AL2023. Updated config:

container_commands:
  01:
    command: "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json -s"
  02:
    command: "sudo systemctl enable amazon-cloudwatch-agent.service"
  03:
    command: "sudo systemctl restart amazon-cloudwatch-agent"

Schaffel

I got this to work, if the new loggroups are created beforehand. Did you ever get it working where it can dynamically create the necessary loggroups? – Brigandine

@StevenLewis if you see the config file above, i am creating a log group for my spring boot logs dynamically. you just need to choose the path of the logs for your app and define it in the config in a similar way. – Schaffel

Answer 3

I've tried Pau Seglar's suggestion of .ebextension config file, but it didn't work for me on Linux 2023 environment.

I have followed logs-streamtocloudwatch-linux.config instead.

.ebextensions/logs-streamtocloudwatch.config:

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json":
    mode: "0644"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/app/current/log/shoryuken.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/current/log/shoryuken.log"]]}`",
                  "log_stream_name": "{instance_id}",
                  "retention_in_days": 7
                }
              ]
            }
          }
        }
      }

container_commands:
  01_append_cloudwatch_agent_config:
    command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json
  02_remove_backup_file:
    command: rm -f /opt/aws/amazon-cloudwatch-agent/etc/custom_logs.json.bak
    ignoreErrors: true

Introgression