Unable to validate the following destination configurations (S3 to SQS)

Question

I am trying to set up a workflow with serverless that creates a new S3 bucket, a new SQS queue and when an object is created in the S3 bucket, puts a messages on the queue and spins up a lambda once there are enough messages on the queue. I have the following in my resources block:

resources:
  Resources:
    AnalyticsQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "my-queue"
    S3EventQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      DependsOn: AnalyticsQueue
      Properties:
        PolicyDocument:
          Id: SQSPolicy
          Statement:
            - Effect: Allow
              Action: sqs:SendMessage:*
              Resource: !Ref AnalyticsQueue
        Queues:
          - !GetAtt AnalyticsQueue.Arn
    AnalyticsBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: "my-bucket"
        NotificationConfiguration:
          QueueConfigurations:
            - Event: s3:ObjectCreated:*
              Queue: !GetAtt AnalyticsQueue.Arn

When I try to deploy this I receive the following error:

An error occurred: AnalyticsBucket - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: E2A1F8BD6BEE6EF4;).

Some googling and I found that the issue is in the NotificationConfiguration block on the AnalyticsBucket. If I remove that whole sub-block, it deploys just fine but then obviously won't generate messages on the queue when objects get created.

Looking for a way to resolve this.

Heel

Answer 1

You need to add an inline "access" policy like this to the SQS queue:

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SQS:SendMessage"
      ],
      "Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::<my-bucket-name>"
        },
        "StringEquals": {
          "aws:SourceAccount": "bucket-owner-account-id"
        }
      }
    }
  ]
}

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html

NOTE: placeholder like <region> in the JSON need to be replaced.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Allow Amazon S3 to use this key",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:<region>:<account-id>:<key-alias>",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "<account-id>"
        },
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::<bucket-name>"
        }
    },
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<account-id>:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    }
  ]
}

Your queue can also use Server-Side Encryption. In which case you also need to add a policy to the Customer KMS (MUST be Customer KMS key, default AWS keys will not work; please read Why aren’t Amazon S3 event notifications delivered to an Amazon SQS queue that uses server-side encryption?)

Twenty

Thanks - same error and solution for configuring SNS Topic as the event notification destination too. – Ultramontane

Answer 2

A lot of AWS configuration allows you to connect services and they fail at runtime if they don't have permission, however S3 notification configuration does check some destinations for access.

In this case, you haven't allowed S3 to send messages to SQS.

It should be something like:

  PolicyDocument:
    Id: SQSPolicy
    Statement:
    - Sid: SQSEventPolicy
      Effect: Allow
      Principal: "*"
      Action: SQS:*
      Resource: "*"
      Condition:
        ArnLike:
          aws:SourceArn: arn:aws:s3:::*

Terrarium

Answer 3

This tutorial helped me to solve the Unknown Error API response: https://www.youtube.com/watch?v=S7SFw8mMMTM

As a summary what needed to be done is to delete this part in the access policy:

"StringEquals": {
          "aws:SourceAccount": "bucket-owner-account-id"

so the access policy will look like this:

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SQS:SendMessage"
      ],
      "Resource": "arn:aws:sqs:Region:account-id:queue-name",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
        }
      }
    }
  ]
}

Hope it helps

Abomasum

Answer 4

This errror may be predominantly due to encryption enabled in the sqs queue. The solution is either disable encryption in sqs or else use an encryption key with proper permissions to key the encrypt/decrypt s3 notification.

Tizes

Answer 5

I tried to create the event manually in the AWS console and got the same problem. It seems this problem occurs if we register an event to encrypted SQS. I tried to disable the SQS encryption to solve this problem. Maybe for encrypted SQS, further configuration is needed.

Pennyworth

This walkthrough metntions it. The KMS key needs an attached policy: ``` { "Version": "2012-10-17", "Id": "example-ID", "Statement": [ { "Sid": "example-statement-ID", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "*" } ] } ``` – Mv

Answer 6

I was facing the same issue to give cross account access to publish S3 update message to my SQS. It can be referenced from https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html

I have a CDK package and the following worked for me And we can set up the policy.

    this.<QueueName>.addToResourcePolicy(
        new iam.PolicyStatement({
            effect: Effect.ALLOW,
            principals: [new SecureServicePrincipal('s3.amazonaws.com')],
            actions: ['SQS:SendMessage'],
            resources: [this.<QueneName>.queueArn],
            conditions: {
                ArnLike: {
                    'aws:SourceArn': <S3ARN>,
                },
                StringEquals: {
                    'aws:SourceAccount': <S3AccountId>,
                }
            }
        })
    );

Colvin

Answer 7

"StringEquals": { "aws:SourceAccount": "bucket-owner-account-id"

Deleting this string worked for me, if you need additional assistance please see the following Youtube: https://www.youtube.com/watch?v=S7SFw8mMMTM

Bullring

Answer 8

I was getting similar error as OP. For me, the issue was with CloudFormation stack Execution Role.

I was trying to Update S3 (adding sending Event Notification to Lambda Function), but Stack Failed to create with similar Error Message.

Solution was to create Role for CloudFormation with Full permissions to Lambda and S3 (of course, only part of these were needed, but it proves that issue was not in definitions in my case, but rather in Stack Execution - be aware).

Other thing that can help (Occured for me, same Error message): Add an Resource-Based Permission to S3, so it can execute that function:

  CFName:
    Type: AWS::Lambda::Permission
    Properties:
         Action: lambda:*
         FunctionName: !Ref <YOUR_FUNC_NAME>
         Principal: s3.amazonaws.com
         SourceArn: !Join 
                  - ''
                  - - 'arn:aws:s3:::'
                    - <BUCKET_NAME>

Memorial

Answer 9

The AWS:SourceAccount should be your Account ID (Top right of the page, clicking on your username)

"StringEquals": {
  "AWS:SourceAccount": "XXXXXXXXXXX"
}   

See AWS Documentation

Wellgrounded

the documentation that you have linked is for SNS nor SQS – Spermatic

Answer 10

While I would definitely check the policy on your SQS queue as the first thing, the second thing would be to open your browser's developer tools and look at the actual request that the console sends back.

In our case, the response identified a specific queue that no longer existed (even though the AWS console UI just showed the generic "some error happened" message.

Additionally, the queue causing the error was not the queue we were trying to target with our new rule - it turns out that all the event notifications on a bucket are managed as one package. So "adding" a new event notification is really updating the existing set of event notifications.

If your existing set of event notifications contains references to Queues or other targets that no longer exist, (in our case a QA environment that had been deleted long ago), the API request to update the the set of notifications fails, because it references queues that no longer exists. The solution is to first delete the event notification rules that reference non-existent targets, then you can add a new rule.

Adler