AWS Elastic Beanstalk: Custom Cloudwatch Logs not showing despite IAM permissions and custom config

Question

I've been having issues setting up custom log file streaming to CloudWatch from my AWS Elastic Beanstalk project. For reference, I've tried the suggestions in "AWS Elastic Beanstalk: Add custom logs to CloudWatch?" to no avail. Essentially, when I download the logs off the host, the log files I want are located at /var/log/containers/api-1c080332ba3f-stdouterr.log, /var/log/containers/nginx-a5057f87f4cf-stdouterr.log, and /var/log/containers/web-0a2e0762e8f0-stdouterr.log (where the numbers change on each update).

I added a new custom log config file in .ebextensions/log.config (following this for reference):

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/containers/nginx-stdouterr.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/containers/nginx-stdouterr.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/containers/nginx*

      [/var/log/containers/web-stdouterr.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/containers/web-stdouterr.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/containers/web*

      [/var/log/containers/api-stdouterr.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/containers/api-stdouterr.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/containers/api*

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd

I've additionally added this policy to the service and ec2 roles:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Finally, I don't see any errors in /var/logs/awslogs.log.

Is there any other piece I am missing? Looked through the official documentation with no luck so far.

Sprinkler

Were you able to resolve this issue ? – Shepp

Answer 1

Instead of adding your own policy to the instance role, try adding the CloudWatchAgentServerPolicy managed policy.

I also removed the /etc/awslogs/awslogs.conf file definition when mine worked.

Finally, the agent only seems to create log groups in CloudWatch when lines are written to the log files. Make sure the files you are collecting are getting written to and see if the log groups get created. Good luck!

Pleasantry

Answer 2

I think you should create an option_settings within .ebextensions

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

In addition, for those who had migrated to amazon-linux-2023, the refered .ebextensions/log.config is not working yet.

I have solved it with this configuration

Jeseniajesh