Reuse Redis connections for NodeJS Lambda function

Question

I have written the following Lambda function:

exports.handler = (event, context, callback) => {
    const redis = require('redis');
    const redis_client = redis.createClient({
        host: 'hostname',
        port: 6379
    });

    redis_client.set("foo", "bar");

    redis_client.get("foo", function(err, reply) {
        redis_client.quit();
        callback(null, reply);
    });
};

This works fine. However, I would like to reuse the Redis connection between Lambda invocations to improve performance. In theory this would be possible by moving the createClient() to outside of the handler. However, because of the "redis_client.quit()" line, that connection is killed. If I do not quit the client, the Lambda function will time out however.

What is the proper way to reuse a Redis in NodeJS when using AWS Lambda?

Peppi

Have a look at the context.callbackWaitsForEmptyEventLoop, it comes with other drawbacks, but your lambda should not time out. How much time is taking the redis.createClient ? – Fante

Answer 1

To reuse the Redis client connection define it as a global outside the handler.

const redis = require('redis');
const redis_client = redis.createClient({
    host: 'hostname',
    port: 6379
});

exports.handler = (event, context, callback) => {
    redis_client.set("foo", "bar");

    redis_client.get("foo", function(err, reply) {
        redis_client.unref();
        callback(null, reply);
    });
};

Redoubtable

unref() is an experimental method so, I suggest to avoid until it will be ready for production. – Franchescafranchise

Answer 2

As you already mentioned your approach is correct way but you have to keep in mind that your Redis instance has connection limits, example AWS Elasticache maxclients is set to 65000. AWS currently allows executing 1k Lambdas in parallel so be careful with the external connections.

Currently, there is no silver bullet for Lambda external DB connections. One possible solution would be to create internal web API service which handles the communication between the DB.

Florist

Answer 3

To avoid the timeout, set callbackWaitsForEmptyEventLoop to false in the AWS Context of your Lambda. This way you do not have to close the redis connection and your lambda will not wait for it (or other connections, i.e. db connections) to close before returning:

https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html

Doyledoyley

Answer 4

Rather than creating the connection for each invocation. A better pattern would be to declare a variable in the global scope then check if empty inside the handler.

This way you are able to establish the connection once per execution environment.

const redis = require('redis');

let redisClient;

exports.handler = (event, context, callback) => {
  context.callbackWaitsForEmptyEventLoop = false;

  if (!redisClient) {
    redisClient = redis.createClient({
      host: 'hostname',
      port: 6379
    });
  }

  redisClient.set("foo", "bar");

  ...
}

https://docs.aws.amazon.com/lambda/latest/operatorguide/global-scope.html https://github.com/sequelize/sequelize/issues/4938#issuecomment-245211042

Situated