Lambda function can't access Secrets Manager

Question

I wrote a lambda function to access a database so the first step is to get secrets from AWS Secrets Manager. I have a private VPC as well as subnets, NAT Gateway, and security group associated with the lambda function. I also have secretsmanager.Secret.grantRead(lambda_exec_role) so the lambda should have access to Secrets Manager.

For some reason when I test it in API Gateway, I got "errno": "ETIMEDOUT" and "code": "NetworkingError" in CloudWatch. And from the printed log I had in the API, getting secrets was failed.

I also tried to add a VPC endpoint for Secrets Manager as in here, but still got the same error.

Appreciated if anyone here could help me with this or give some hints.

Many thanks!

Fanion

When you test your lambda function in console, not using API gateway, does it work? Or nothing works? – Banquer

@Banquer No it doesn't work either. It gave "errorType": "NetworkingError", "errorMessage": "connect ETIMEDOUT" msg. – Fanion

So your lambda-vpc integration is incorrect. You have to check your vpc endpoints for SSM, nat gateway setup, route tables, security groups. – Banquer

Rather than creating a Nat Gateway, adding Secrets Manager into VPC might help. docs.aws.amazon.com/secretsmanager/latest/userguide/… – Grillage

If you can rule out connectivity as the issue, the default time out of 3 seconds may not be enough for the first execution. Mine took 7 seconds on the first run and .2 thereafter. Increasing the time out to 8 seconds was the fix. – Woald

Answer 1

I had trouve with a lambda getting secret content too.

They are several things you can try :

#1 Make sure you have permission to get the secret value, I'll give you mine for a working configuration :

• Allow:secretsmanager:GetSecretValue on your secret
• Allow:secretsmanager:DescribeSecret on your secret
• Allow:secretsmanager:ListSecrets on all ressources

#2 I had trouble too with my VPC and subnets. If misconfugred, you won't be able to call Secret Manager API.

• Switch to no VPC for your lambda and check if you can get your secret OK. If it works then it means you have a problem with your VPC/subnet configuration.
• Check your subnet configuration :
  • On public subnet, you can configure a specific endpoint for secret manager, though I couldn't make it work, don't know why.
  • On private subnet, you nea to configure a NAT Gateway to be able to call for Secret Manager API.

Hoping it could help someone someday. :)

Bradlybradman

My connection to Secret Manager was timing out with no error. What unblock me was I needed to create a VPC endpoint to the secret manager. Thanks for pointing me in the right direction. – Console

After a lot of trial error, my issue involved also adding "secretsmanager:GetResourcePolicy" and "secretsmanager:ListSecretVersionIds" along with the policies you listed. – Everybody

I had to switch to using private subnet working but public subnet not working for the lambda – Tonetic

Answer 2

In case anybody else arrives at this question when troubleshooting issues with the Secrets Manager lambda extension I wanted to share my experience when implementing using the Nodejs18.x runtime (brand new at this time). I was taking advantage of the new top-level await from this new runtime and hoped to retrieve my secret from SM on init (outside the handler). All my attempts failed and I was receiving a response from the extension

not ready to serve traffic, please wait

Moving the http call (to the extension's port) inside of my handler function immediately fixed the issue.

Arbogast

Thanks, your comment finally lead me into the right direction. While we don't use top-level await so far, I had a created a Promise for the lookup on top-level (to be awaited later in async code), which of course started to resolve immediately. Which means the HTTP call was indeed made during module loading phase, not inside of the handler. – Retinol

Spent three days adjusting my project for ESM required by top-level awaits only to stumble upon this response after getting "not ready to serve traffic, please wait". If I can't fetch a secret connection string in a top-level await then I cannot connect to the database, and then what's the point of it all? – Motorboat

Answer 3

I also had a similar problem and what worked for me is to add kms:Decryptto the IAM Policy. Because according to the official AWS documentation:

If the secret is encrypted using a customer-managed key instead of the Amazon Web Services managed key aws/secretsmanager , then you also need kms:Decrypt permissions for that key. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value

The IAM policy attached to Lambda should then look something like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
        }
    ]
}

You might want to limit Resources to something more specific based on your requirements. Hope it also works for you.

Damaraland

Answer 4

By the nature of the error - network timeout - I would assume you couldn't get your secrets even if you had permissions.

Since you're dealing with a Lambda inside a VPC, please note there are two ways of placing your Lambda in a VPC:

• in a public subnet: in this case, your Lambda CANNOT have access to the internet, so your only option would be to configure a VPC endpoint for the Secrets Manager, or
• in a private subnet: in this case, your private subnet can have route 0.0.0.0/0 traffic to a NAT Gateway, then to an Internet Gateway (through a public subnet). This gives you the option to access Secrets Manager through its public endpoint (i.e. through the internet), or, again, through a VPC endpoint.

Sarpedon

This solution worked in my case where my lambda residing on a private subnet had to access secrets from secrets manager. I had to create a vpc endpoint on the subnet where the lambda resides for secrets manager access. – Tunny

Answer 5

To chime in on this, my solution required two steps

• Create a VPC Endpoint for Secrets Manager (I followed this process here: https://aws.amazon.com/blogs/security/how-to-connect-to-aws-secrets-manager-service-within-a-virtual-private-cloud/)
• Attach the policy to the Executor Role just as @frida-schenker noted earlier.

Tittle

Answer 6

Here's how I got it working in serverless.yml.

AWS Reference: https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html#vpc-endpoint

The following yml appears under the path resources.Resources.YourVPCEndpointNameHere:

# Provides access from the VPC to Secrets Manager
# See https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html#vpc-endpoint
Type: AWS::EC2::VPCEndpoint
Properties:
  VpcEndpointType: Interface
  ServiceName: com.amazonaws.#{AWS::Region}.secretsmanager
  PrivateDnsEnabled: true

  # Reference your VPC here
  VpcId: !Ref EC2VPC

  # Reference your subnet ids here
  SubnetIds:
    - !Ref EC2SubnetA
    - !Ref EC2SubnetB
    - !Ref EC2SubnetC

  # Reference your security group(s) here
  SecurityGroupIds:
    - !Ref EC2SecurityGroup

Tip: You'll need the plugin serverless-pseudo-parameters to make #{AWS::Region} work.

Alleluia