Elastic beanstalk not streaming custom log to cloudwatch [ amazon linux 2 ]

Question

I am unable to send custom logs to Cloudwatch. I have created /opt/aws/amazon-cloudwatch-agent/etc/shoryuken.json which contains

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/app/current/log/shoryuken.log",
            "log_group_name": "/aws/elasticbeanstalk/Demo-env/var/app/current/log/shoryuken.log",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

Then appending this change with this command /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/shoryuken.json -s

Outout:

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
I! Detecting run_as_user...
I! Trying to detect region from ec2
D! [EC2] Found active network interface
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded

I can see the amazon-cloudwatch-agent.toml has the appended config

I have stoped and started /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl

Still not receiving any logs to cloudwatch.

Mulloy

Answer 1

I am able to fix this after much digging. I had to fix IAM policy and needed to restart cloudwatch agent with systemd command

IAM policy attached to my ec2 profile user

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups"
            ],
            "Resource": "*"
        }
    ]
}

command to restart amazon-cloudwatch-agent

sudo /bin/systemctl restart amazon-cloudwatch-agent.service

Mulloy

That will work but you're not using least privilege here. For resource, don't add all. Instead pick an arn or list of arns that map to specific log groups. What you're specifiying is almost getting to CloudWatchFullAccess, which is a managed policy. – Hi