How do I manage MongoDB connections in a Node.js web application?

Question

I'm using the node-mongodb-native driver with MongoDB to write a website.

I have some questions about how to manage connections:

1. Is it enough using only one MongoDB connection for all requests? Are there any performance issues? If not, can I setup a global connection to use in the whole application?

2. If not, is it good if I open a new connection when request arrives, and close it when handled the request? Is it expensive to open and close a connection?

3. Should I use a global connection pool? I hear the driver has a native connection pool. Is it a good choice?

4. If I use a connection pool, how many connections should be used?

5. Are there other things I should notice?

Health

@IonicãBizãu, sorry, I haven't use nodejs for a long time that I haven't see it. Thanks for your comment~ – Health

Connection class and Promise Global Variable – Expansible

Answer 1

The primary committer to node-mongodb-native says:

You open do MongoClient.connect once when your app boots up and reuse the db object. It's not a singleton connection pool each .connect creates a new connection pool.

So, to answer your question directly, reuse the db object that results from MongoClient.connect(). This gives you pooling, and will provide a noticeable speed increase as compared with opening/closing connections on each db action.

Philpot

Link to MongoClient.connect() mongodb.github.io/node-mongodb-native/driver-articles/… – Repose

This is the correct answer. The accepted answer is very wrong as it says to open a connection pool for each request and then close it after doing so. Terrible architecture. – Nightdress

I've added an example here: groups.google.com/d/msg/node-mongodb-native/mSGnnuG8C1o/… – Emergence

This is a right answer. My god imagine that I have to open and close each time I do something it would be 350K per hour just for my inserts! It's like attacking my own server. – Naucratis

@Max: How do you share db object between modules? Use global.db = db; or is there any other way? – Stater

@Cracker: If you have express application, you can save db object into req.db with this middleware: github.com/floatdrop/express-mongo-db – Madonnamadora

I don't think this library really solves the problem completely if you're after something manual. The middleware appears to provide all requests access to a single connection instance (good), but looking at the source I don't believe this binds any kind of early-disconnect query management to express. Please correct me if wrong, see related post here. – Zeeland

@Stater I created a module for this exact purpose github.com/toymachiner62/mongo-factory – Anna

When would you close the connection? Why would you even want to close this other than on stopping the app server? If you stop your app server does the connection automatically close anyway? – Rickey

if there are multiple database.. should i open a connection for each database all together. If not, Is it OK to open and close when required? – Jeffiejeffrey

@AmanGupta in that case you would need to open a connection for each database. – Mound

@ThomasBratt How can we handle the case when the mongo connection dies at a later stage after being connected? Trying to reuse the db object that results from MongoClient.connect() would fail in that scenario untill the node application is restarted. – Mound

@Mound sorry, I'm no longer active in Node.js. – Ewaewald

Is this answer still correct with the latest SDK? Now connect returns a client (not a db). Should we still assume the same pattern? Call connect only once, and close only once? – Bisectrix

This is what I have been doing for the last couple of years, and seems to be 🆗 AFAIK. LMK if you find something bad about it. – Behnken

Related as issue: github.com/mongodb-developer/nodejs-quickstart/issues/15 – Kure

Is there any difference between saving connection in a variable or a collection in a variable? – Maffick

Answer 2

Open a new connection when the Node.js application starts, and reuse the existing db connection object:

/server.js

import express from 'express';
import Promise from 'bluebird';
import logger from 'winston';
import { MongoClient } from 'mongodb';
import config from './config';
import usersRestApi from './api/users';

const app = express();

app.use('/api/users', usersRestApi);

app.get('/', (req, res) => {
  res.send('Hello World');
});

// Create a MongoDB connection pool and start the application
// after the database connection is ready
MongoClient.connect(config.database.url, { promiseLibrary: Promise }, (err, db) => {
  if (err) {
    logger.warn(`Failed to connect to the database. ${err.stack}`);
  }
  app.locals.db = db;
  app.listen(config.port, () => {
    logger.info(`Node.js app is listening at http://localhost:${config.port}`);
  });
});

/api/users.js

import { Router } from 'express';
import { ObjectID } from 'mongodb';

const router = new Router();

router.get('/:id', async (req, res, next) => {
  try {
    const db = req.app.locals.db;
    const id = new ObjectID(req.params.id);
    const user = await db.collection('user').findOne({ _id: id }, {
      email: 1,
      firstName: 1,
      lastName: 1
    });

    if (user) {
      user.id = req.params.id;
      res.send(user);
    } else {
      res.sendStatus(404);
    }
  } catch (err) {
    next(err);
  }
});

export default router;

Source: How to Open Database Connections in a Node.js/Express App

Mona

This creates one database connection...if you want to utilize pools you have to create/close on each use – Mackie

Never heard of app.locals before, but I'm glad you introduced me to them here – Enterogastrone

Helped me a lot! I make the mistake to create/close DB connection for every request, the perfomance of my app dropped down with this. – Salubrious

I need to thank you for my single most impactful performance optimization to this point. – Portamento

Answer 3

Here is some code that will manage your MongoDB connections.

var MongoClient = require('mongodb').MongoClient;
var url = require("../config.json")["MongoDBURL"]

var option = {
  db:{
    numberOfRetries : 5
  },
  server: {
    auto_reconnect: true,
    poolSize : 40,
    socketOptions: {
        connectTimeoutMS: 500
    }
  },
  replSet: {},
  mongos: {}
};

function MongoPool(){}

var p_db;

function initPool(cb){
  MongoClient.connect(url, option, function(err, db) {
    if (err) throw err;

    p_db = db;
    if(cb && typeof(cb) == 'function')
        cb(p_db);
  });
  return MongoPool;
}

MongoPool.initPool = initPool;

function getInstance(cb){
  if(!p_db){
    initPool(cb)
  }
  else{
    if(cb && typeof(cb) == 'function')
      cb(p_db);
  }
}
MongoPool.getInstance = getInstance;

module.exports = MongoPool;

When you start the server, call initPool

require("mongo-pool").initPool();

Then in any other module you can do the following:

var MongoPool = require("mongo-pool");
MongoPool.getInstance(function (db){
    // Query your MongoDB database.
});

This is based on MongoDB documentation. Take a look at it.

Listel

Update since 5.x: var option = { numberOfRetries : 5, auto_reconnect: true, poolSize : 40, connectTimeoutMS: 30000 }; – Jeweller

@Yaki, could you please take a look at this problem ?#72439919 – Patterson

Answer 4

Manage mongo connection pools in a single self contained module. This approach provides two benefits. Firstly it keeps your code modular and easier to test. Secondly your not forced to mix your database connection up in your request object which is NOT the place for a database connection object. (Given the nature of JavaScript I would consider it highly dangerous to mix in anything to an object constructed by library code). So with that you only need to Consider a module that exports two methods. connect = () => Promise and get = () => dbConnectionObject.

With such a module you can firstly connect to the database

// runs in boot.js or what ever file your application starts with
const db = require('./myAwesomeDbModule');
db.connect()
    .then(() => console.log('database connected'))
    .then(() => bootMyApplication())
    .catch((e) => {
        console.error(e);
        // Always hard exit on a database connection error
        process.exit(1);
    });

When in flight your app can simply call get() when it needs a DB connection.

const db = require('./myAwesomeDbModule');
db.get().find(...)... // I have excluded code here to keep the example  simple

If you set up your db module in the same way as the following not only will you have a way to ensure that your application will not boot unless you have a database connection you also have a global way of accessing your database connection pool that will error if you have not got a connection.

// myAwesomeDbModule.js
let connection = null;

module.exports.connect = () => new Promise((resolve, reject) => {
    MongoClient.connect(url, option, function(err, db) {
        if (err) { reject(err); return; };
        resolve(db);
        connection = db;
    });
});

module.exports.get = () => {
    if(!connection) {
        throw new Error('Call connect first!');
    }

    return connection;
}

Dichromate

Better yet, you could just get rid of the connect() function and have the get() function check to see if connection is null and call connect for you if it is. Have get() always return a promise. This is how I manage my connection and it works great. This is a use of the singleton pattern. – Savage

@Savage While that approach does provide a more streamlined API it does have two drawbacks. The first being that there is no defined way to check for connection errors. You would have to handle that inline everywhere when ever you call get. I like to fail early with database connections and generally I won't let the app boot with out a connection to the database. The other issue is throughput. Because you don't have an active connection you might have to wait for a bit longer on the first get() call which you won't have control over. Could skew your reporting metrics. – Dichromate

@Dichromate The way I structure applications/services is to typically retrieve a configuration from the database on startup. In this way, the application will fail to start if the database is inaccessible. Also, because the first request is always on startup, there's no issue with the metrics with this design. Also do rethrow a connection exception with in once place in the singleton with a clear error it can't connect. Since the app needs to catch database errors when using the connection anyway, this doesn't lead to any extra inline handling. – Savage

@Dichromate How would you handle the case when the mongo connection dies at a later stage after being connected? All calls to get() would fail in that scenario untill the node application is restarted. – Mound

Hi @Ayan. It's important to note that here that when we call get() we are getting a connection pool not a single connection. A connection pool, as it's name implies is a logical collection of database connections. If there are no connections in the pool the driver will attempt to open one. Once that connection is open it's used and returned to the pool. The next time the pool is accessed this connection maybe reused. The nice thing here is the pool will manage our connections for us so if a connection is dropped we might never know as the pool will open a new one for us. – Dichromate

It does not make any sense.because db.get() need to wait for connection.So still we need to use await or callback() or Promise... ( db.connect() .then(() => db.get()) ). even mongodb also throw error if you use db object before connection...you don't have to throw error manually... – Festinate

Hi @Festinate I think you might be confusing two different error cases. In the first error case I throw if the connection object does not exist. This error would only happen when a connection has not even been attempted thus the null check. In the second error case we can not connect to the database when we attempt to. In this error case we do let the mongo client handle the error for us by rejecting the promise returned by the connect function. – Dichromate

Answer 5

If you have Express.js, you can use express-mongo-db for caching and sharing the MongoDB connection between requests without a pool (since the accepted answer says it is the right way to share the connection).

If not - you can look at its source code and use it in another framework.

Madonnamadora

Answer 6

You should create a connection as service then reuse it when need.

// db.service.js
import { MongoClient } from "mongodb";
import database from "../config/database";

const dbService = {
  db: undefined,
  connect: callback => {
    MongoClient.connect(database.uri, function(err, data) {
      if (err) {
        MongoClient.close();
        callback(err);
      }
      dbService.db = data;
      console.log("Connected to database");
      callback(null);
    });
  }
};

export default dbService;

my App.js sample

// App Start
dbService.connect(err => {
  if (err) {
    console.log("Error: ", err);
    process.exit(1);
  }

  server.listen(config.port, () => {
    console.log(`Api runnning at ${config.port}`);
  });
});

and use it wherever you want with

import dbService from "db.service.js"
const db = dbService.db

Natica

If mongo couldn't connect, MongoClient.close() gives an error. But a good solution for original problem. – Receivable

Answer 7

I have been using generic-pool with redis connections in my app - I highly recommend it. Its generic and I definitely know it works with mysql so I don't think you'll have any problems with it and mongo

https://github.com/coopernurse/node-pool

Haynes

Mongo already does connection pooling in the driver, I have, however mapped my mongo connections into an interface that matches the node-pool, this way all my connections follow the same pattern, even though in the case of mongo, the cleanup doesn't actually trigger anything. – Havildar

Answer 8

I have implemented below code in my project to implement connection pooling in my code so it will create a minimum connection in my project and reuse available connection

/* Mongo.js*/

var MongoClient = require('mongodb').MongoClient;
var url = "mongodb://localhost:27017/yourdatabasename"; 
var assert = require('assert');

var connection=[];
// Create the database connection
establishConnection = function(callback){

                MongoClient.connect(url, { poolSize: 10 },function(err, db) {
                    assert.equal(null, err);

                        connection = db
                        if(typeof callback === 'function' && callback())
                            callback(connection)

                    }

                )



}

function getconnection(){
    return connection
}

module.exports = {

    establishConnection:establishConnection,
    getconnection:getconnection
}

/*app.js*/
// establish one connection with all other routes will use.
var db = require('./routes/mongo')

db.establishConnection();

//you can also call with callback if you wanna create any collection at starting
/*
db.establishConnection(function(conn){
  conn.createCollection("collectionName", function(err, res) {
    if (err) throw err;
    console.log("Collection created!");
  });
};
*/

// anyother route.js

var db = require('./mongo')

router.get('/', function(req, res, next) {
    var connection = db.getconnection()
    res.send("Hello");

});

Bullfight

Answer 9

If using express there is another more straightforward method, which is to utilise Express's built in feature to share data between routes and modules within your app. There is an object called app.locals. We can attach properties to it and access it from inside our routes. To use it, instantiate your mongo connection in your app.js file.

var app = express();

MongoClient.connect('mongodb://localhost:27017/')
.then(client =>{
  const db = client.db('your-db');
  const collection = db.collection('your-collection');
  app.locals.collection = collection;
});
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              // view engine setup
app.set('views', path.join(__dirname, 'views'));

This database connection, or indeed any other data you wish to share around the modules of you app can now be accessed within your routes with req.app.locals as below without the need for creating and requiring additional modules.

app.get('/', (req, res) => {
  const collection = req.app.locals.collection;
  collection.find({}).toArray()
  .then(response => res.status(200).json(response))
  .catch(error => console.error(error));
});

This method ensures that you have a database connection open for the duration of your app unless you choose to close it at any time. It's easily accessible with req.app.locals.your-collection and doesn't require creation of any additional modules.

Fernandes

Answer 10

Best approach to implement connection pooling is you should create one global array variable which hold db name with connection object returned by MongoClient and then reuse that connection whenever you need to contact Database.

1. In your Server.js define var global.dbconnections = [];

2. Create a Service naming connectionService.js. It will have 2 methods getConnection and createConnection. So when user will call getConnection(), it will find detail in global connection variable and return connection details if already exists else it will call createConnection() and return connection Details.

3. Call this service using <db_name> and it will return connection object if it already have else it will create new connection and return it to you.

Hope it helps :)

Here is the connectionService.js code:

var mongo = require('mongoskin');
var mongodb = require('mongodb');
var Q = require('q');
var service = {};
service.getConnection = getConnection ;
module.exports = service;

function getConnection(appDB){
    var deferred = Q.defer();
    var connectionDetails=global.dbconnections.find(item=>item.appDB==appDB)

    if(connectionDetails){deferred.resolve(connectionDetails.connection);
    }else{createConnection(appDB).then(function(connectionDetails){
            deferred.resolve(connectionDetails);})
    }
    return deferred.promise;
}

function createConnection(appDB){
    var deferred = Q.defer();
    mongodb.MongoClient.connect(connectionServer + appDB, (err,database)=> 
    {
        if(err) deferred.reject(err.name + ': ' + err.message);
        global.dbconnections.push({appDB: appDB,  connection: database});
        deferred.resolve(database);
    })
     return deferred.promise;
} 

Pacificas

Answer 11

In case anyone wants something that works in 2021 with Typescript, here's what I'm using:

import { MongoClient, Collection } from "mongodb";

const FILE_DB_HOST = process.env.FILE_DB_HOST as string;
const FILE_DB_DATABASE = process.env.FILE_DB_DATABASE as string;
const FILES_COLLECTION = process.env.FILES_COLLECTION as string;

if (!FILE_DB_HOST || !FILE_DB_DATABASE || !FILES_COLLECTION) {
  throw "Missing FILE_DB_HOST, FILE_DB_DATABASE, or FILES_COLLECTION environment variables.";
}

const client = new MongoClient(FILE_DB_HOST, {
  useNewUrlParser: true,
  useUnifiedTopology: true,
});

class Mongoose {
  static FilesCollection: Collection;

  static async init() {
    const connection = await client.connect();
    const FileDB = connection.db(FILE_DB_DATABASE);
    Mongoose.FilesCollection = FileDB.collection(FILES_COLLECTION);
  }
}


Mongoose.init();

export default Mongoose;

I believe if a request occurs too soon (before Mongo.init() has time to finish), an error will be thrown, since Mongoose.FilesCollection will be undefined.

import { Request, Response, NextFunction } from "express";
import Mongoose from "../../mongoose";

export default async function GetFile(req: Request, res: Response, next: NextFunction) {
  const files = Mongoose.FilesCollection;
  const file = await files.findOne({ fileName: "hello" });
  res.send(file);
}

For example, if you call files.findOne({ ... }) and Mongoose.FilesCollection is undefined, then you will get an error.

Disfranchise

Answer 12

Using below method you can easily manage as many as possible connection

var mongoose = require('mongoose');


//Set up default mongoose connection
const bankDB = ()=>{
    return  mongoose.createConnection('mongodb+srv://<username>:<passwprd>@mydemo.jk4nr.mongodb.net/<database>?retryWrites=true&w=majority',options);
    
}

bankDB().then(()=>console.log('Connected to mongoDB-Atlas bankApp...'))
       .catch((err)=>console.error('Could not connected to mongoDB',err));
       
//Set up second mongoose connection
const myDB = ()=>{
    return  mongoose.createConnection('mongodb+srv://<username>:<password>@mydemo.jk4nr.mongodb.net/<database>?retryWrites=true&w=majority',options);
   
}
myDB().then(()=>console.log('Connected to mongoDB-Atlas connection 2...'))
       .catch((err)=>console.error('Could not connected to mongoDB',err));

module.exports = { bankDB(), myDB() };

Sputter

Answer 13

npm i express mongoose

mongodb.js

const express = require('express');
const mongoose =require('mongoose')
const app = express();

mongoose.set('strictQuery', true);
mongoose.connect('mongodb://localhost:27017/db_name', {
    useNewUrlParser: true, 
    useUnifiedTopology: true
})
.then(() => console.log('MongoDB Connected...'))
.catch((err) => console.log(err))

app.listen(3000,()=>{ console.log("Started on port 3000 !!!") })

node mongodb.js

Dygert