AWS Lambda Container destroy event

Question

When to release connections and cleanup resources in lambda. In normal Node JS application we do use the hook

process.on('exit', (code) => {
    console.log(`About to exit with code: ${code}`);
});

However this doesn't work on AWS Lambda. Resulting the Mysql connection in sleep mode. We don't have enough resource for such active connections. None of the AWS documentation specify a way to achieve this.

How to receive stop event of AWS Lambda container ?

Gains

To address database connections specifically, RDS Lambda Proxies are a good solution for this problem. – Cinda

Answer 1

The short answer is that there is no such event to know when the container is stopped.

UPDATE: I've not used this library, but https://www.npmjs.com/package/serverless-mysql appears to try to solve just this problem.

PREVIOUS LONG ANSWER:

The medium answer: after speaking with someone at AWS about this I now believe you should scope database connections at the module level so they are reused as long as the container exists. When your container is destroyed the connection will be destroyed at that point.

Original answer:

This question touches on some rather complicated issues to consider with AWS Lambda functions, mainly because of the possibility of considering connection pooling or long-lived connections to your database. To start with, Lambda functions in Node.js execute as a single, exported Node.js function with this signature (as you probably know):

exports.handler = (event, context, callback) => {
    // TODO implement
    callback(null, 'Hello from Lambda');
};

The cleanest and simplest way to handle database connections is to create and destroy them with every single function call. In this scenario, a database connection would be created at the beginning of your function and destroyed before your final callback is called. Something like this:

const mysql = require('mysql');

exports.handler = (event, context, callback) => {
  let connection = mysql.createConnection({
    host     : 'localhost',
    user     : 'me',
    password : 'secret',
    database : 'my_db'
  });

  connection.connect();

  connection.query('SELECT 1 + 1 AS solution', (error, results, fields) => {
    if (error) {
      connection.end();
      callback(error);
    }
    else {
      let retval = results[0].solution;
      connection.end();
      console.log('The solution is: ', retval);
      callback(null, retval);
    }
  });
};

NOTE: I haven't tested that code. I'm just providing an example for discussion.

I've also seen conversations like this one discussing the possibility of placing your connection outside the main function body:

const mysql = require('mysql');

let connection = mysql.createConnection({
  host     : 'localhost',
  user     : 'me',
  password : 'secret',
  database : 'my_db'
});

connection.connect();

exports.handler = (event, context, callback) => {
  // NOTE: should check if the connection is open first here
  connection.query('SELECT 1 + 1 AS solution', (error, results, fields) => {
    if (error) {
      callback(error);
    }
    else {
      let retval = results[0].solution;
      console.log('The solution is: ', retval);
      callback(null, retval);
    }
  });
};

The theory here is this: because AWS Lambda will try to reuse an existing container after the first call to your function, the next function call will already have a database connection open. The example above should probably check the existence of an open connection before using it, but you get the idea.

The problem of course is that this leaves your connection open indefinitely. I'm not a fan of this approach but depending on your specific circumstances this might work. You could also introduce a connection pool into that scenario. But regardless, you have no event in this case to cleanly destroy a connection or pool. The container process hosting your function would itself be killed. So you'd have to rely on your database killing the connection from it's end at some point.

I could be wrong about some of those details, but I believe at a high level that is what you're looking at. Hope that helps!

Derail

I'm using the second approach. That is why i need to close the connection. First approach is compromising the performance. It is establishing connection at each request that is very costly in our application. Is there any way to close the connection with second approach ? – Gains

@ViswanathLekshmanan I am not aware of any mechanisms that would allow you to reliably close and destroy connections outside of the function execution itself. We do not connect to our database directly from Lambda, but instead host an API for internal use on ECS (Docker containers) where we have complete control. – Derail

Thanks for the info. – Gains

I've minus'ed this, because the "answer" doesn't provide any solution on when to close the connection. – Bahaism

@ViktorMolokostov the question is regarding how to receive stop event of a Lambda container. The answer is that there is no such event. – Derail

I've done some recent performance evaluation on differences between the first and second method in late 2020, and can confidently say that in the simple case of opening a DB connection to v12 Postgres hosted on AWS RDS, reading some data and closing the connection, there is very little performance lost between the first and second methods. Use the first method in this answer, open the connection before you read, and close the connection in a finally before the function exits. Do a quick performance check and done! @ViktorMolokostov – Channa

@TimFulmer I think it's dangerous to draw conclusions from comparing a handful of execution times using Method A (create-destroy in function scope) vs Method B (module scope). To really get a handle on what's happening I suspect you need to do load testing where an increasing number of executions happen concurrently. I would expect Method B to perform better at scale. Also, look at the DB metrics during these tests to see how it performs in each scenario. You may also need to try throttling function executions to limit concurrency so you don't have too many connections to the database. – Derail

@ToddPrice right. I am only speaking to the time taken to open and close a new connection for each request, with a correctly sized db. There may be details about the SQL run which does not scale concurrently, as you state. And if the db runs out or RAM, that's it. Concurrent connections can eat up RAM, depending on the queries run. That said, I am not seeing any reason to assume create-destroy is negatively impacting performance on it's own. I have many high volume apps using this pattern in production today. Database connection create times are not the performance bottleneck in any of them. – Channa

Keeping the database connection outside is a clever idea but make sure that one instance running does not have its connection closed by the time it is about to make a query. This gets really tricky to debug this issue. As far as the load test is concerned opening and closing connection each time increases parallel db connections count to more 250%. This is another factor to look out for other than just performance. – Apocarp

Answer 2

I believe Lambda Extensions provide a means for capturing finalization logic. Here’s a Stack Overflow Answer that discusses this option. It points to this section of the AWS documentation. Presumably, you could shutdown db connections inside a finalization callback.

I’m curious about this as it appears to be a cleaner shutdown method, however I have not yet tested it.

Chlor