AWS managed Ariflow doesn't fetch log

Question

I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch.

I am getting the following error in the log of Air flow:

*** Reading remote log from Cloudwatch log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Could not read remote logs from log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.

Denison

Answer 1

AWS has a troubleshooting guide for this specific case.

*** Reading remote log from Cloudwatch log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.Could not read remote logs from log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.

https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs

I would check the following:

1. Verify that you enabled task logs at the INFO level for your environment.
2. Check if the MWAA Execution role has CloudWatch read access policy attached.
3. Eventually try adding apache-airflow[amazon] provider to the MWAA Requirements file.

Tessitura

apache-airflow[amazon] does adding this help resolve the issue? we have tried the first 2 – Disseminule

Answer 2

In my case the issue was because of encryption. Check if your MWAA uses the same KMS key as your CloudWatch Log Groups.

If MWAA is created with custom KMS key this key is also used for CloudWatch Log Groups encryption. When you delete and re-create MWAA (with new KMS key) Log Groups are not touched and remain encrypted with old key. And they are not accessible for MWAA any more.

Two ways to fix it:

1. Delete Log Groups, create new Log Groups with the same name and using KMS key which is used by MWAA.
2. Don't use KMS key when you create MWAA. In this case, AWS managed key aws/airflow will be used automatically. It should help to avoid dependency on particular KMS key.

Kind

Answer 3

It was an issue related to role and policies.

MWAA Execution role didn't have the required CloudWatch access policy attached to it.

Denison

can u shed further light on this. we are facing similar issue and are unable to figure it out. – Disseminule

@PriyadarshanMohanty in my case this was an issue: github.com/idealo/terraform-aws-mwaa/issues/42 – Stalinabad

For me re-deploying terraform from us-east-1 to eu-central-1 also solved the issue. As far as I remember, without KMS encryption logs were available at us-east-1 as well. – Idzik

Hi @AntonBryzgalov, is this the backend script that's used in MWAA? I think the issue is more related to the worker getting killed before the task gets scheduled on the worker node. – Disseminule

Hi @PriyadarshanMohanty, no, this is just an externally maintained Terraform module for MWAA. Not related to MWAA internals, but due to region mismatch I have had the same issue as in the post. – Stalinabad

What is the required CloudWatch access policy ? – Gisborne

Hi, i am facing issues with only some tasks , but some tasks are working fine. Any similar issues? – Cutcheon

why not just add the policy? this comment is less than useful – Anachronism

Answer 4

In My case, this issue happened because the KMS used for encrypting the S3 bucket and MWAA was missing following in the Key policy

"kms:GenerateDataKey*",

for principal

AWS": "arn:aws:iam:::root"

Kolva

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Moia