ECS Fargate Scheduled Task not running

Question

I'm trying to setup a scheduled task with ECS Fargate but I cannot figure out why it is not running. I can confirm the task works correctly using RunTask but when I try to trigger it on a schedule all I get is a bunch of 'FailedInvocations' with no explanation.

I do know though that the rule is being triggered so this is a good sign. See the screenshot below:

But everytime it is triggered there is just a 'FailedInvocation'. Here's the scheduling rule:

And the default permissions on the ecsEventRole with just ecs:runTask:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

My hunch says that this ecsEventsRole doesn't have enough permissions. Should I try to give it the ones that ecsTaskExecutionRole has?

Thanks

EDIT: This is now supported in us-east-1 region. See comments.

Multiped

Any updates? I'm at exactly the same spot and stuck – Cinereous

Nope. I'm still stuck :( Given up on all options. – Multiped

I talked to AWS support, its not supported yet. No ETA – Cinereous

Brutal. Thanks Marc – Multiped

@MarcYoung exactly what is not supported? – Dikdik

Scheduled Fargate tasks are not supported. Only EC2 scheduled ecs tasks – Cinereous

Fargate tasks are now supported: aws.amazon.com/about-aws/whats-new/2018/08/… – Osseous

Works in us-east-1, but unavailable in ca-central-1. Assuming there are other problem regions. – Multiped

Answer 1

Although it's been over an year, AWS still don't have a proper way to see invocation logs.

As you already know we can invoke tasks by RunTask manually, so does Scheduled Task.

The only difference is that Scheduled Task is triggered by CloudWatch rules.

We can easily see invoke logs in CloudTrail Event history, go there and filter events with Event name: RunTask and select the time range you want to check, find the event and click View Event, you'll see the error code and response.

Peccant

After I combined top 2 answer, added ecs:RunTask and "iam:ListInstanceProfiles", "iam:ListRoles", "iam:PassRole" to the cloudwatch event iam role, finally the task run successfully. – Shastashastra

This is actually the correct answer. Errors in the actual config aside, this problem was exacerbated by the logs being hidden. I had no idea Cloudtrail even existed, never mind that it ingests internal AWS events. – Materfamilias

Answer 2

I ran into a similar issue where regular ECS Scheduled Tasks were not running.

I finally resolved it by adding an additional policy to ecsEventsRole which allows CloudWatch Events to pass IAM roles to ECS Tasks:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListInstanceProfiles",
                "iam:ListRoles",
                "iam:PassRole"
            ],
            "Resource": "*"
        }
    ]
}

Rammer

I think this is hitting a few people because the first bit of documentation (docs.aws.amazon.com/AmazonECS/latest/developerguide/…) says 'You must add iam:PassRole permissions for any task role overrides to the CloudWatch IAM role', but it turns out that the task needs it to use a role already applied (explained in a different area of the documentation docs.aws.amazon.com/AmazonECS/latest/developerguide/… - 'If your scheduled tasks require the use of the task execution role or a task role override, then you must add iam:PassRole permissions'). – Cahn

In addition to ecs:RunTask, I only needed iam:PassRole on both the Task Role and Task Execution role resources. – Afresh

Answer 3

Here is a possible workaround: use a lambda function as target for the cloudwatch rule and create the task in the lambda function code.

Here is an example code for the lambda function: https://lobster1234.github.io/2017/12/03/run-tasks-with-aws-fargate-and-lambda/

The links describes how to pack the new boto version with the lambda function but this is not necessary anymore since AWS already updated the lambda boto version to 1.4.8

I've tested and it works.

Manuel

Thats what I was trying to avoid but ended up with – Cinereous

I ended up using this. Probably the most reasonable solution presently. – Multiped

Answer 4

When defining a Rule in CloudFormation, I had to ensure that the Target RoleArn property referenced ecsEventsRole:

Targets:
- Id: !Sub ${AWS::StackName}-cron-test-1
  Arn: arn:aws:ecs:eu-west-1:<account id>:cluster/fargate-cluster
  RoleArn: !Join
    - ':'
    - - "arn:aws:iam"
      - ""
      - !Ref "AWS::AccountId"
      - "role/ecsEventsRole"

And where ecsEventsRole defined the AmazonEC2ContainerServiceEventsRole policy

Sines

Answer 5

Have you tried using the aws cli and running aws events put-rule followed by aws events put-targets --rule <value> --targets <value> instead? I was having a similar problem, and using the (recent version of) the aws cli worked for me.

Here's a sample:

aws events put-rule --name "DailyLambdaFunction" --schedule-expression "cron(0 9 * * ? *)"

Followed by the below command all in one line:

aws events put-targets --rule cli-RS-rule --targets '{"Arn":"arn:aws:ecs:1234/cluster/clustername","EcsParameters":{"LaunchType": "FARGATE","NetworkConfiguration": {"awsvpcConfiguration": {"AssignPublicIp": "ENABLED", "SecurityGroups": [ "sg-id1233" ], "Subnets": [ "subnet-1234" ] }},"TaskCount": 1,"TaskDefinitionArn": "arn:aws:ecs:1234:task-definition/taskdef"},"Id": "sampleID111","RoleArn": "arn:aws:iam:1234:role/eventrole"}'

Yacov

make sure to edit the security group, subnet and account ID strings before running it - they're just placeholders in the command i used above. – Yacov

Answer 6

I had similar problem with ECS fargate task scheduler. I've managed to fix it by adding below policy to the task IAM role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask"
            ],
            "Resource": [
                "arn:aws:ecs:*:548503722878:task-definition/test-services-20200813124716"
            ],
            "Condition": {
                "ArnLike": {
                    "ecs:cluster": "arn:aws:ecs:*:548503722878:cluster/test-cluster"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": [
                "*"
            ],
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "ecs-tasks.amazonaws.com"
                }
            }
        }
    ]
}

Aidoneus

Answer 7

I found that all the answers are missing one very important point:

The role which is used as "ecsEventsRole" (or "Target Role" if you look at it from the ECS->Scheduled_tasks) must have a Trust to the Events service in order to be able to run the task:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "events.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }
]

}

Also (as previously mentioned) it should have the permissions to run the tasks - easily achieved by adding the predefined AWS policy:

AmazonEC2ContainerServiceEventsRole

Mensural

Answer 8

I don't know how it works, but it did. I did turn ON [auto-assign a public IP] while scheduling task to ecs. I was stuck in this error for hours.

Grounds