Lambda does not have permission to access the ECR image

Question

With the recent release of Docker Images for Lambda functions, I've decided to try out this functionality using CloudFormation.

So, the lambda below considers a docker image stored in Elastic Container Registry, with permissions to access the image following the examples in the documentation.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: lambda-docker-image

Globals:
  Function:
    Timeout: 180

Resources:
  DockerAsImage:
    Type: AWS::Serverless::Function 
    Properties:
      FunctionName: DockerAsImage
      ImageUri: ??????????????.dkr.ecr.us-west-2.amazonaws.com/????:latest
      PackageType: Image
      Policies: 
        - Version: '2012-10-17' 
          Statement:
            - Effect: Allow
              Action: 
                - ecr:*
                - ecr-public:*
                - sts:GetServiceBearerToken
              Resource: "*"
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: post

I'm using sam to deploy the template in us-west-2 with

sam deploy -t template.yaml --capabilities "CAPABILITY_NAMED_IAM" --region "us-west-2" --stack-name "lambda-docker-example" --s3-bucket "my-bucket" --s3-prefix "sam_templates/lambda-docker-example" --force-upload  --no-confirm-changeset

However, just after the IAM Role is succesfuly created, the Lambda function fails to create with the following error

Lambda does not have permission to access the ECR image. Check the ECR permissions. (Service: AWSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException;

even though the role has access to any ecs resource. Another way I've tried is to create a separate role and assigned it to lambda through Role: !GetAtt Role.Arn, this approach doesn't work too.

Freeman

Do your IAM user/role has permissions to ECR? For using Image-based lambda function, you as a user/role which creates the function needs ECR permissions as explained here. Can you check that? – Subequatorial

@Subequatorial I had the same issue and your advice solved it! – Camenae

@alexyz78 Thanks for letting me know. If it will for the OP I can will provide more info in the answer. – Subequatorial

@Subequatorial yes indeed those two permissions solve the issue, additionally to push a docker image to ecr it was necessary ecr: InitiateLayerUpload . Thank you very much Marcin, AWS docs doesn't tend to be an easy guideline. Feel free to share your answer. – Freeman

@MiguelTrejo Thanks, answer added. – Subequatorial

Answer 1

Based on the comments.

To use image-based lambdas, it is the IAM user/role that requires ECR permissions, not the function itself. From docs:

Make sure that the permissions for the AWS Identity and Access Management (IAM) user or role that creates the function contain the AWS managed policies GetRepositoryPolicy and SetRepositoryPolicy.

In addition to the two permissions listed above, the ecr: InitiateLayerUpload is also needed.

Subequatorial

Hmm, we seemed to also need LambdaECRImageRetrievalPolicy. – Generalization

Should be ECR resource policy or IAM role policy? I have added to IAM role policy and it does not work – Cameraman

Answer 2

For a lambda in account 222222222222 to use an ECR image in 11111111111 then you need to follow https://aws.amazon.com/blogs/compute/introducing-cross-account-amazon-ecr-access-for-aws-lambda/

The most important IAM part is to set the following Repository policy on the 11111111111 repo:

      RepositoryPolicyText:
        Version: "2012-10-17"
        Statement:
          - Sid: CrossAccountPermission
            Effect: Allow
            Action:
              - ecr:BatchGetImage
              - ecr:GetDownloadUrlForLayer
            Principal:
              AWS:
                - arn:aws:iam::222222222222:root
          - Sid: LambdaECRImageCrossAccountRetrievalPolicy
            Effect: Allow
            Action:
              - ecr:BatchGetImage
              - ecr:GetDownloadUrlForLayer
            Principal:
              Service: lambda.amazonaws.com
            Condition:
              StringLike:
                aws:sourceArn:
                  - arn:aws:lambda:us-east-1:222222222222:function:*

Septic

Answer 3

You must to add the following policy to your User and the Role that will be asociated to the AWS Lambda. This policy enables ECR actions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecr:SetRepositoryPolicy",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:CompleteLayerUpload",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:UploadLayerPart",
                "ecr:ListImages",
                "ecr:InitiateLayerUpload",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetRepositoryPolicy",
                "ecr:PutImage"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        }
    ]
}

St

Answer 4

I was facing the same issues with all the required AWS lambda policies in place. What helped me was adding permissions in ECR

{
      "Sid": "LambdaECRImageRetrievalPolicy",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ]
}   

Although aws also says if lambda has policies (ecr:getRepositoryPolicy and ecr:setRepositoryPolicy) then we don't need to add permission in ECR lambda automatically does that.

If the Amazon ECR repository does not include these permissions, Lambda adds ecr:BatchGetImage and ecr:GetDownloadUrlForLayer to the container image repository permissions. Lambda can add these permissions only if the Principal calling Lambda has ecr:getRepositoryPolicy and ecr:setRepositoryPolicy permissions.

Reference #1, #2

Amnesia

Answer 5

Had a similar issue, when I had ECR in Account A and needed to create a Lambda in Account B.

The solution was adding the following to ECR repo in account A :

Reference

{
  "Version": "2008-10-17",
  "Statement": [  
    {
      "Sid": "LambdaECRImageRetrievalPolicyCrossAccount",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": [
        "ecr:BatchGetImage",
        "ecr:DeleteRepositoryPolicy",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:SetRepositoryPolicy"
      ],
      "Condition": {
        "StringLike": {
          "aws:sourceArn": "arn:aws:lambda:us-east-2:{account_id}:function:*"
        }
      }
    },
    {
      "Sid": "CrossAccountPermission",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{account_id}:root"
      },
      "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ]
    }
  ]
}

Misconduct

Answer 6

My conribution does not really answer the OP's problem but it might help someone looking for a solution to their problem - if it is the same case as mine.

I found out through some trail and error (not with ANY help from CloudTrail or any docs from AWS) that the problem was not with Lambda's permissions to GET the images from ECR, but to be able to set ECR's policy (ecr:SetRepositoryPolicy) because of an SCP explicit deny.

If you want to see if this is your case, try to access the repo and change the policy, if you can't, this is it. There is no use in trying anything else.

Mangonel