AWS CloudWatch Rule returns FailedInvocation with AWS batch as Target

Question

Hi I've scheduled a Cloudwatch rule in order to run every Wednesday at 14.15 GTM by having as target an AWS Batch, which always returns FailedInvocation. I'm seeing the FailedInvocation event from associated metrics

However there are no logs regarding the error, I cannot understand the problem.

I've followed this tutorial: https://docs.aws.amazon.com/batch/latest/userguide/batch-cwe-target.html I'm stucked here from hours any suggestion?

CONFIGURATIONS

The AWS batch target is configured as is:

• job Queue= arn:..
• job Definition = arn:...
• Job Name = name

The role associated to the target has the following policies:

• arn:aws:iam::aws:policy/service-role/AWSBatchServiceEventTargetRole

  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "batch:SubmitJob"
          ],
          "Resource": "*"
      }
    ]
    }
  

• arn:aws:iam::216314997889:role/awsInvokeActionOnEc2

  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "cloudwatch:Describe*",
              "ec2:Describe*",
              "ec2:RebootInstances",
              "ec2:StopInstances",
              "ec2:TerminateInstances"
          ],
          "Resource": "*"
      }
  ]
  }
  

• and Trust relationships

  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
   ]
  }
  

Hydroplane

Any Suggestion? – Hydroplane

Hi, did you ever find an answer for this? Thanks. – Lindsay

HI, yes by using CloudTrail I've found that I was using a wrong ARN job definition. – Hydroplane

Answer 1

Enable CloudTrail to find out the FailedInvocation reason in its logs. I agree going through CloudTrail to find out the failure reason is terrible. But for now, that's all there is. Faced the same issue and found out the Input

Imperfection

I've found your suggestion very helpful, btw for the ones who read to find informations about submitting job status via cloudwatch rule: Go to CloudTrail -> Event History and filter by Event Source: batch.amazonaws.con you will see appearing rows with EventName=SubmitJob, click on View Event. it will open a json object, scroll down until you found "responseElements and errorCode to view the problem – Hydroplane

You saved my day guys thanks :) – Townsman

Answer 2

If anyone ever encounters FailedInvocations from event rules targeting Cloudwatch log groups, this is most likely due to the absence of a "Cloudwatch log resource policy" permitting the AWS Events service to create Cloudwatch logs. If you create the rule through the console, there should be an appropriate one automatically provisioned. You can check whether you have one provisioned:

aws logs describe-resource-policies

If you already have an appropriate Cloudwatch log resource policy configured, you should see something like:

{
    "resourcePolicies": [
        {
            "policyName": "TrustEventsToStoreLogEvents",
            "policyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"TrustEventsToStoreLogEvent\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":[\"events.amazonaws.com\",\"delivery.logs.amazonaws.com\"]},\"Action\":[\"logs:PutLogEvents\",\"logs:CreateLogStream\"],\"Resource\":\"arn:aws:logs:eu-central-1:1234567890:log-group:/aws/events/*:*\"}]}",
            "lastUpdatedTime": 1641611871623
        }
    ]
}

However, if you've configured your rules with Terraform (maybe even Cloudformation), then this will probably not automatically be provisioned.

Here's an example Terraform excerpt to provision a policy matching the one auto-configured through the console:

data "aws_iam_policy_document" "events_delivery_logs_write_logs" {
  statement {
    sid = "TrustEventsToStoreLogEvent"

    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = ["arn:${data.aws_partition.current.partition}:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/events/*:*"]

    principals {
      identifiers = [
        "events.amazonaws.com",
        "delivery.logs.amazonaws.com"
      ]
      type = "Service"
    }
  }
}

resource "aws_cloudwatch_log_resource_policy" "events_delivery_logs_write_logs" {
  policy_document = data.aws_iam_policy_document.events_delivery_logs_write_logs.json
  # This is the standard name this is utilized when creating a CW event rule -> CW log group through the console
  policy_name = "TrustEventsToStoreLogEvents"
}

Infra resources:

• Cloudformation resource
• Terraform resource

Suribachi

Answer 3

If you are looking for the reason your invocations are failing, see the other answers UNLESS you're trying to implement AWS::Events::Rule and you're seeing failed invocations. The following answer may resolve the issue and negate to need to find these non-existent logs.

Cloudwatch failedinvocation error no logs available

Windy