Handling timeouts with Node.js and mongodb

Question

I am currently testing how some code stands up against the following scenario:

• Node.js application is started and successfully establishes a connection to mongodb
• After it has successfully setup a connection, the mongodb server dies and all subsequent requests fail

To do this I have got the following code which makes use of the official driver (found here: https://github.com/mongodb/node-mongodb-native) :

MongoClient.connect('mongodb://localhost:27017/testdb', function(err, db) {
app.get('/test', function(req, res) {
    db.collection('users', function (err, collection) {
        console.log(err);
        if (err) {
            // ## POINT 1 ##
            // Handle the error
        }
        else {
            collection.find({ 'username': username }, { timeout: true }).toArray(function(err, items) {
                console.log(err);
                if (err) {
                    // ## POINT 2 ##
                    // Handle the error
                }
                else {
                    if (items.length > 0) {
                        // Do some stuff with the document that was found
                    }
                    else {
                        // Handle not finding the document
                    }
                }
            }); 
        }
    });
});

});

As the mongodb server is no longer running when the request is being handled, I'd made the assumption that at either the points which I have labelled ## POINT 1 ## or ## POINT 2 ##, it would return an error indicating a timeout; this however, isn't the case.

I have tried a number of different settings (including one you can see here that explicitly allows the cursor to timeout), however I cannot seem to enable it in any way. In every configuration I've tried Node.js will simply keep waiting for the find() operation to callback and it never does.

If I start the Node.js application before running mongodb, it catches the error in the connect callback fine, but if the connection dies after that it doesn't seem to handle it in any way.

Is there a setting I am missing or is there no way to detect connections being terminated after they've been established?

Edit: just to be clear, the username variable used in the find method is actually declared in my full code, the code I've put in this post is a cut down version to illustrate the structure and error checking.

Chaparajos

This is a good question: db.collection (point #1 above) will not return an error when the db connection is lost. collection.find (point #2 above) will eventually return an error, but it will be after more than 1 min. I played with the connectionTimeoutMS and the socketTimeoutMS settings in the connection string, but the behavior did not change. – Rafaellle

Hey @JesusRuiz, I had left the find call running for about 15-20 minutes I think it was but it never called back. I had previously tried changing the timeout properties but as you mentioned they didn't seem to have any effect! I'm not sure if I've discovered a bug in the driver or if there's something I'm still not seeing :( – Chaparajos

Answer 1

UPD:
Based on this post, looks like they've deployed fix that will do the same as what we do here. Not sure if this is already within npm (15.10.13). https://github.com/mongodb/node-mongodb-native/issues/1092#ref-commit-2667d13

After some investigation I've managed to understand what is going on there:
Every time you call any method to deal with database (find, update, insert, etc.) it creates cursor, that has own ID and registers itself to EventEmitter of Db for being called back later. In meantime it registers itself to _notReplied object within same CallBackStore.

But once connection is closed, I couldn't locate anything that would iterate through _notReplied cursors and would trigger them with errors or any logic with timers (it still might be somewhere there). So I've managed to write small work around, that does force triggers cursors with error when DB emits close event:

new mongodb.Db('testdb', new mongodb.Server('localhost', 27017, { }), { safe: true }).open(function (err, db) {
  if (!err) {
    db.on('close', function() {
      if (this._callBackStore) {
        for(var key in this._callBackStore._notReplied) {
          this._callHandler(key, null, 'Connection Closed!');
        }
      }
    });

    // ...

  } else {
    console.log(err)
  }
});

I recommend using first approach instead of MongoClient. Reasons are few: for example when you close connection and then call .find it will properly trigger error in callback, while with MongoClient it won't.

If you are using MongoClient:

MongoClient.connect('mongodb://localhost:27017/testdb', function(err, db) {
  if (!err) {
    db.on('close', function() {
      if (this._callBackStore) {
        for(var key in this._callBackStore._notReplied) {
          this._callHandler(key, null, 'Connection Closed!');
        }
      }
    });

    // ...

  } else {
    console.log(err);
  }
});

What this will do? Once connection is closed, it will iterate through All _notReplied cursors and trigger events for them with error Connection Closed!.

Test case:

items.find({ }).toArray(function(err, data) {
  if (!err) {
    console.log('Items found successfully');
  } else {
    console.log(err);
  }
});
db.close();

That will force close database connection and trigger close event that you handle earlier and will make sure that cursor will be closed.

UPD: I've added Issue on GitHub: https://github.com/mongodb/node-mongodb-native/issues/1092 we'll see what they say regarding this.

Herson

Excellent find! I'll give this a try when I get back from work and let you know how it turns out. Thanks. – Chaparajos

Hey, I gave this a try but I'm not sure it meets what I was looking for. I only realised as trying it that this won't be able to handle scenarios where for example the database server crashes before executing find() as the close event will occur before hand, leaving the find() call stuck with no callback. I think the only way to do it would be to just check the connection state before executing find() and hope that within those few milliseconds everything is OK! – Chaparajos

I couldn't actually get your snippet to work by the way, I placed it after db had been instantiated and was open, however there was no _callBackStore object in this, the closest I could find is this.serverConfig._callBackStore._notReplied – Chaparajos

You are using MongoClient, which is not straight DB class, so do I've updated my answer for your case. – Herson

Regarding issue with .find check comment I've wrote regarding MongoClient (after first option). – Herson

I tried the revised code but it still errors in the callback for the close event, likewise it doesn't seem to trigger the callback of find() if calling it when the connection is closed :( I've uploaded a full code sample here for you to see in case I've done something different: gist.github.com/rastating/e616b24f0abcee72ac5a – Chaparajos

I can't see where you close database connection. Could you specify the way you do it? – Herson

Sorry I should have mentioned - I am closing the database by actually killing the mongod process, as to try and emulate the database connection being closed from something other than the node application itself – Chaparajos

Then how do you make sure that you have closed it right in a middle of find? As I can see in your code lack of console logs to track that. In your code you assign route /test, so how do you close database right in a middle of that request is happening? – Herson

Just tried - close event is triggered when connection is closed externally. And if there is any hanging cursors - they will be closed. Make sure you test it properly, as in your code you need to make /test request and as quick as possible turn off database - which is practically difficult to simulate. – Herson

Sorry, the problem I came across with _callBackStore not existing was when testing to see how it responds when there are no cursors open. Presumably _callBackStore is explicitly set to undefined when there are no callbacks pending? – Chaparajos

I've edited answer to check for its existence. It is weird, as it should exist as it is created when Db class is initialized. – Herson

I came across the same issue. It very frustrating issue and definetely should be addressed in the driver and not by users inventing some tricks to get around it. – Deracinate

@Deracinate it has been already addressed, please read first paragraph in the answer. You might consider updating your dependencies to later versions. – Herson

Answer 2

I had the same problem, and found this page from google. But your choosed answer didn't resolve the problem and it is as same as you, this._callBackStore can't use

but i tried to wrap the Mongo, and it seems work fine

var MongoClient = require('mongodb').MongoClient;

var mongo = {};
mongo.init = function() {
  MongoClient.connect('mongodb://localhost:27017/testdb', function(err, db) {
    if (err) {
      mongo.DB = '';
    } else {
      mongo.DB = db;
    }
    db.on('close', function() {
      mongo.DB = '';
    });
    db.on('reconnect', function() {
      mongo.DB = db;
    });
  }
}
                      
mongo.getdb = function(callback) {
    if (mongo.DB) {
      callback(null, mongo.DB);
    } else {
      callback('can not connect to db', null);
    }
}
module.exports = mongo;

firstly start server and init() it

and then you can require it and use

mongo.getdb(function(err, db) {
  if (err) {
    console.log(err);
  } else {
    db.collection('user').find({'xxx':'xxx'}).toArray(function(err, items) {
      console.log(items);
    });
  }
});

Occupant

Answer 3

After some further investigation, it seems that you can't specify "offline" timeouts such as in the scenario outlined above. The only timeout that can be specified is one which informs the server to timeout the cursor after 10 minutes of inactivity, however as in the scenario above the connection to the server is down this does not work.

For reference, I found the information here: https://github.com/mongodb/node-mongodb-native/issues/987#issuecomment-18915263 by who I believed to be one of the main contributors to the project.

Chaparajos

Link is bad, maybe they changed the location. – Deracinate

Answer 4

I'm making api with Hapi and Mongodb (w/o mongoose). Features:

1. Start responding to API request only if mongo db is available
2. Stop responding if mongo dies during cycle
3. Re-start when mongo available again
4. Keep single connection for all requests

Combining some ideas from other answers and this post https://productbuilder.wordpress.com/2013/09/06/using-a-single-global-db-connection-in-node-js/ my approach is this:

server.js

Utilities.initializeDb(() => {
    server.start((err) => {
        if (err) throw err;
        console.log('Server running at:', server.info.uri);
    });
}, () => {
    server.stop((err) => {
        if (err) throw err;
        console.log('Server stopped');
    });
});

Utilities.js

"use strict";

const MongoClient = require('mongodb').MongoClient;
const MongoUrl = 'mongodb://localhost:27017/db';

export const Utilities = {
    initializeDb: (next, onCrash) => {

        const ConnectToDatabase = (params) => {
            MongoClient.connect(MongoUrl, (err, db) => {
                if (err !== null) {
                    console.log('#t4y4542te Can not connect to mongo db service. Retry in 2 seconds. Try #' + params.retry);
                    console.error(err);
                    setTimeout(() => {
                        ConnectToDatabase({retry: params.retry + 1});
                    }, 2000);
                } else {

                    db.on('close', () => {
                        onCrash();
                        console.log('#21df24sf db crashed!');
                        ConnectToDatabase({retry: 0});
                    });
                    global.db = global.db || db;
                    next();
                }
            });
        };

        ConnectToDatabase({retry: 0});

    }
};

I'm exporting db connection to global space. It feels like not best solution, but I had projects where db connection was passed as param to all modules and that sucked more. Maybe there should be some modular approach where you import db connection where you need it, but in my situation i need it almost everywhere, I would have to write that include statement in most files. This API is pointless w/o connection to db, so I think it might be best solution even if I'm against having something flying magically in global space..

Mallory