AWS Lambda can't connect to RDS instance, but I can locally?

Question

I am trying to connect to my RDS instance from a lambda. I wrote the lambda locally and tested locally, and everything worked peachy. I deploy to lambda, and suddenly it doesn't work. Below is the code I'm running, and if it helps, I'm invoking the lambda via a kinesis stream.

'use strict';

exports.handler = (event, context, handlerCallback) => {
    console.log('Recieved request for kinesis events!');
    console.log(event);
    console.log(context);

    const connectionDetails = {
        host:     RDS_HOST,
        port:     5432,
        database: RDS_DATABASE,
        user:     RDS_USER,
        password: RDS_PASSWORD
    };

    const db = require('pg-promise')({promiseLib: require('bluebird')})(connectionDetails);

    db
            .tx(function () {
                console.log('Beginning query');

                return this.query("SELECT 'foobar'")
                           .then(console.log)
                           .catch(console.log)
                           .finally(console.log);
            })
            .finally(() => handlerCallback());
};

Here is the logs from cloud watch if it helps:

START RequestId: *********-****-****-****-********* Version: $LATEST 
2016-05-31T20:58:25.086Z    *********-****-****-****-*********  Recieved request for kinesis events! 
2016-05-31T20:58:25.087Z    *********-****-****-****-*********  { Records:  [ { kinesis: [Object], eventSource: 'aws:kinesis', eventVersion: '1.0', eventID: 'shardId-000000000000:**********************************', eventName: 'aws:kinesis:record', invokeIdentityArn: 'arn:aws:iam::******************:role/lambda_kinesis_role', awsRegion: 'us-east-1', eventSourceARN: 'arn:aws:kinesis:us-east-1:****************:stream/route-registry' } ] } 
2016-05-31T20:58:25.283Z    *********-****-****-****-*********  { callbackWaitsForEmptyEventLoop: [Getter/Setter], done: [Function], succeed: [Function], fail: [Function], logGroupName: '/aws/lambda/apiGatewayRouteRegistry-development', logStreamName: '2016/05/31/[$LATEST]******************', functionName: 'apiGatewayRouteRegistry-development', memoryLimitInMB: '128', functionVersion: '$LATEST', getRemainingTimeInMillis: [Function], invokeid: '*********-****-****-****-*********', awsRequestId: '*********-****-****-****-*********', invokedFunctionArn: 'arn:aws:lambda:us-east-1:*************:function:apiGatewayRouteRegistry-development' } 
END RequestId: *********-****-****-****-********* 
REPORT RequestId: *********-****-****-****-*********    Duration: 20003.70 ms   Billed Duration: 20000 ms Memory Size: 128 MB   Max Memory Used: 22 MB   
2016-05-31T20:58:45.088Z *********-****-****-****-********* Task timed out after 20.00 seconds

Layla

Did you enable VPC access for your Lambda function? – Ghiselin

@MarkB nope. I have the dropdown set to "No VPC" – Layla

So you have Publicly Accessible enabled on your RDS instance, and you have the RDS instance's security group open to the world? – Ghiselin

@MarkB Yes, and I can access RDS form my laptop directly. The whole lambda works perfectly fine on my local machine. – Layla

20 seconds would seem like enough time, but does it work if you bump up the Lambda timeout temporarily to 60s? – Shutin

@Shutin Yes, I had it at 60 seconds at one point. No change – Layla

You really should connect this to the VPC that (hopefully) your RDS instance is in. Otherwise your RDS instance has to allow access from 0.0.0.0/0 -- which it really sounds like is not currently the case, but if it is, it's a terrible idea from a security perspective. – Radiancy

@Michael-sqlbot you are correct, having it in the same VPC would be ideal, but we are just trying to bootstrap this for internal development, we aren't releasing anything yet. We will lock down security before it goes live. And the RDS instance is public available right now. – Layla

@MarkB @Michael-sqlbot, I finally got AWS support response to point out that the RDS security group was indeed private to a specific IP. This doesn't make sense as I never configured that, and I could access the database from my local machine and elastic beanstalk. I added 0.0.0.0/0 to the security group and now the lambda can connect. Thanks for your help guys! – Layla

Answer 1

@MarkB @Michael-sqlbot were correct in the comments, it was a security group issue.

I finally got AWS support response to point out that the RDS security group was indeed private to a specific IP. This doesn't make sense as I never configured that, and I could access the database from my local machine and elastic beanstalk. I added 0.0.0.0/0 to the security group and now the lambda can connect. Thanks for your help guys!

Layla

Hello, Can you elaborate? Where and to what did you add the 0.0.0.0/0? – Dodiedodo

But adding 0.0.0.0/0 to your inbound in the security group meaning opening the connection to the world. Don't you afraid of that? – Dislodge

@Dislodge it was for development and testing purposes. Later it got locked down to the correct security group. Yes, opening the connection to the world would not be a great idea for production use. – Layla

So, did you eventually add your lambda to the VPC? – Collaborative

@Layla AWS automatically configures your public IP as Inbound in the security group at the moment of the RDS instance creation. That's why the code was working from your machine, but nowhere else. Same happened to me! :) – Letters

Answer 2

Here is how I fixed this issue.

When you create a DB instance, you are asked to select VPC. Even if you select default values, it takes the public IP of your system as default inbound IP. Lambda function, on the other hand, has its own IP setting. That's why you can access through any IDE or locally however not through lambda function.

To add Ip restrictions:

1. Go to Security group of your instance. After selecting the default security group, click on it. In the new page, scroll down to find inbound and outbound settings.

2. In inbound setting, click edit. You can change the IP here. (0.0.0.0/0 makes it open to the world)

3. If you add public IP here then IDE or your local connection would work.

4. For lambda function to work, add IP of the lambda function. Go to Lambda function, Network --> VPC --> (if no VPC is selected, select a VPC same as DB function) and note the IP here.

5. Type this IP in inbound settings, this will show auto filler.

Save it and test your lambda function.

Pesticide

would this apply if I set Publicly Accessible to true? – Transversal

You are a life saver. Thanks a lot! – Orb

How about the subnets ? Does it matter what it being selected there ? – Housley

Answer 3

Here's the answer to this with no world wide access.

Allow AWS Lambda to access RDS Database

Repeating Mark's answer:

1. Add vpc access to the lambda function
2. Create a new security group for the lambda
3. Add the RDS security group to the lambda's SG.

Salcido

What is this adding to the answers? – Woolfolk

I like this answer better because the first one opens your RDS instance to the world and the second one doesn't work, or at least it didn't for me. This one is elegant and well referenced. – Institutional

How about the subnets ? Does it matter what it being selected there ? – Housley

Answer 4

For me, the issue was that I wasn't using the ssl:true attribute in the Client setup.

var client = new Client({
      user: RDS_USER,
      host: RDS_HOST,
      database: RDS_DB,
      password: RDS_PASSWORD,
      ssl: true
    });

Hemoglobin