Does mongoDB have reconnect issues or am i doing it wrong?
Asked Answered
G

6

31

I'm using nodejs and a mongoDB - and I'm having some connection issues.

Well, actually "wake" issues! It connects perfectly well - is super fast and I'm generally happy with the results.

My problem: If i don't use the connection for a while (i say while, because the timeframe varies 5+ mins) it seems to stall. I don't get disconnection events fired - it just hangs.

Eventually i get a response like Error: failed to connect to [ * .mongolab.com: * ] - ( * = masked values)

A quick restart of the app, and the connection's great again. Sometimes, if i don't restart the app, i can refresh and it reconnects happily.

This is why i think it is "wake" issues.

Rough outline of code:

I've not included the code - I don't think it's needed. It works (apart from the connection dropout)

Things to note: There is just the one "connect" - i never close it. I never reopen.

I'm using mongoose, socketio.

/* constants */

var mongoConnect = 'myworkingconnectionstring-includingDBname';


/* includes */

/* settings */

/* Schema */

var db = mongoose.connect(mongoConnect);

    /* Socketio */

io.configure(function (){
    io.set('authorization', function (handshakeData, callback) {

    });
});

io.sockets.on('connection', function (socket) {

});//sockets

io.sockets.on('disconnect', function(socket) {
    console.log('socket disconnection')
});

/* The Routing */

app.post('/login', function(req, res){  

});

app.get('/invited', function(req, res){

});

app.get('/', function(req, res){

});

app.get('/logout', function(req, res){

});

app.get('/error', function(req, res){

});

server.listen(port);
console.log('Listening on port '+port);

db.connection.on('error', function(err) {
    console.log("DB connection Error: "+err);
});
db.connection.on('open', function() {
    console.log("DB connected");
});
db.connection.on('close', function(str) {
    console.log("DB disconnected: "+str);
});

I have tried various configs here, like opening and closing all the time - I believe though, the general consensus is to do as i am with one open wrapping the lot. ??

I have tried a connection tester, that keeps checking the status of the connection... even though this appears to say everthing's ok - the issue still happens.

I have had this issue from day one. I have always hosted the MongoDB with MongoLab. The problem appears to be worse on localhost. But i still have the issue on Azure and now nodejit.su.

As it happens everywhere - it must be me, MongoDB, or mongolab.

Incidentally i have had a similar experience with the php driver too. (to confirm this is on nodejs though)

It would be great for some help - even if someone just says "this is normal"

thanks in advance

Rob

Gremlin answered 20/12, 2012 at 20:48 Comment(6)
Not sure about MongoDB, but in most (traditional databases) it's better to open/close the connection on demand (i.e. everytime you need to access it) rather than leaving it open forever. Database drivers are usually optimized to create and manage a pool connections behind the scenes so this (opening/closing) tends to be incredibly fast. I would be curious to hear what people with real MongoDB experience say on this question too.Doucet
Thanks Hector - yeah agreed, only everywhere i read when investigating this says to do it the way i have (unless i understand incorrectly)Gremlin
I get the same problem against both MongoHQ and MongoLab instances--not only from my Node app, but from the mongo command-line client too. MongoHQ support suggested I handle this at a app or driver level by retrying when this happens. It's such a pain to handle this at an app level that I'm surprised this isn't already handled in the mongodb driver or in mongoose.Tramel
has anyone figured this out? I have the exact same configuration and the exact same problem. It's worse on localhost but still occurs in the cloud. You can basically get one session out of a restart. MongoLab seems pretty much not usable to me on Azure at this point :(Unstable
Also having this issue, but inside a private hosting solution. Appears to be related to the Mongo connection and keepAlive.Steinway
Check out my solution below.... Seems solid as a rock now.Gremlin
G
21

Thanks for all the help guys - I have managed to solve this issue on both localhost and deployed to a live server.

Here is my now working connect code:

var MONGO = {
    username: "username",
    password: "pa55W0rd!",
    server: '******.mongolab.com',
    port: '*****',
    db: 'dbname',
    connectionString: function(){
        return 'mongodb://'+this.username+':'+this.password+'@'+this.server+':'+this.port+'/'+this.db;
    },
    options: {
        server:{
            auto_reconnect: true,
            socketOptions:{
                connectTimeoutMS:3600000,
                keepAlive:3600000,
                socketTimeoutMS:3600000
            }
        }
    }
};

var db = mongoose.createConnection(MONGO.connectionString(), MONGO.options);

db.on('error', function(err) {
    console.log("DB connection Error: "+err);
});
db.on('open', function() {
    console.log("DB connected");
});
db.on('close', function(str) {
    console.log("DB disconnected: "+str);
});

I think the biggest change was to use "createConnection" over "connect" - I had used this before, but maybe the options help now. This article helped a lot http://journal.michaelahlers.org/2012/12/building-with-nodejs-persistence.html

If I'm honest I'm not overly sure on why I have added those options - as mentioned by @jareed, i also found some people having success with "MaxConnectionIdleTime" - but as far as i can see the javascript driver doesn't have this option: this was my attempt at trying to replicate the behavior.

So far so good - hope this helps someone.

UPDATE: 18 April 2013 note, this is a second app with a different setup

Now I thought i had this solved but the problem rose it's ugly head again on another app recently - with the same connection code. Confused!!!

However the set up was slightly different…

This new app was running on a windows box using IISNode. I didn't see this as significant initially.

I read there were possibly some issues with mongo on Azure (@jareed), so I moved the DB to AWS - still the problem persisted.

So i started playing about with that options object again, reading up quite a lot on it. Came to this conclusion:

options: {
    server:{
        auto_reconnect: true,
        poolSize: 10,
        socketOptions:{
            keepAlive: 1
        }
    },
    db: {
        numberOfRetries: 10,
        retryMiliSeconds: 1000
    }
}

That was a bit more educated that my original options object i state. However - it's still no good.

Now, for some reason i had to get off that windows box (something to do with a module not compiling on it) - it was easier to move than spend another week trying to get it to work.

So i moved my app to nodejitsu. Low and behold my connection stayed alive! Woo!

So…. what does this mean… I have no idea! What i do know is is those options seem to work on Nodejitsu…. for me.

I believe IISNode uses some kind of "forever" script for keeping the app alive. Now to be fair the app doesn't crash for this to kick in, but i think there must be some kind of "app cycle" that is refreshed constantly - this is how it can do continuous deployment (ftp code up, no need to restart app) - maybe this is a factor; but i'm just guessing now.

Of course all this means now, is this isn't solved. It's still not solved. It's just solved for me in my setup.

Gremlin answered 19/1, 2013 at 20:21 Comment(4)
Rather quirky, this is what did it for me: using mongoose.createConnection instead of connect Anyway, a multitude of thanks.Polyanthus
@Gremlin - the article you linked to seems to be missing it's examples. Do you know of an alternative copy? ThanksNarah
@Narah Sorry, i do not.Gremlin
I have used similar setup to this, due to my "zero downtime deploy" ( duplicate cluster then change DNS ) i end up with hundred of connections to mongodb, which completely slowed down my production environment. Now trying to purge the connections on SIGINT/SIGQUIT, trying to get rid of page faults... mongodb connection hellInadvertent
T
32

UPDATE: Our support article for this topic (essentially a copy of this post) has moved to our connection troubleshooting doc.

There is a known issue that the Azure IaaS network enforces an idle timeout of roughly thirteen minutes (empirically arrived at). We are working with Azure to see if we can't make things more user-friendly, but in the meantime others have had success by configuring their driver options to work around the issue.

Max connection idle time

The most effective workaround we've found in working with Azure and our customers has been to set the max connection idle time below four minutes. The idea is to make the driver recycle idle connections before the firewall forces the issue. For example, one customer, who is using the C# driver, set MongoDefaults.MaxConnectionIdleTime to one minute and it cleared up their issues.

MongoDefaults.MaxConnectionIdleTime = TimeSpan.FromMinutes(1);

The application code itself didn't change, but now behind the scenes the driver aggressively recycles idle connections. The result can be seen in the server logs as well: lots of connection churn during idle periods in the app.

There are more details on this approach in the related mongo-user thread, SocketException using C# driver on azure.

Keepalive

You can also work around the issue by making your connections less idle with some kind of keepalive. This is a little tricky to implement unless your driver supports it out of the box, usually by taking advantage of TCP Keepalive. If you need to roll your own, make sure to grab each idle connection from the pool every couple minutes and issue some simple and cheap command, probably a ping.

Handling disconnects

Disconnects can happen from time to time even without an aggressive firewall setup. Before you get into production you want to be sure to handle them correctly.

First, be sure to enable auto reconnect. How to do so varies from driver to driver, but when the driver detects that an operation failed because the connection was bad turning on auto reconnect tells the driver to attempt to reconnect.

But this doesn't completely solve the problem. You still have the issue of what to do with the failed operation that triggered the reconnect. Auto reconnect doesn't automatically retry failed operations. That would be dangerous, especially for writes. So usually an exception is thrown and the app is asked to handle it. Often retrying reads is a no-brainer. But retrying writes should be carefully considered.

The mongo shell session below demonstrates the issue. The mongo shell by default has auto reconnect enabled. I insert a document in a collection named stuff then find all the documents in that collection. I then set a timer for thirty minutes and tried the same find again. It failed, but the shell automatically reconnected and when I immediately retried my find it worked as expected.

% mongo ds012345.mongolab.com:12345/mydatabase -u *** -p *** 
MongoDB shell version: 2.2.2 
connecting to: ds012345.mongolab.com:12345/mydatabase 
> db.stuff.insert({}) 
> db.stuff.find() 
{ "_id" : ObjectId("50f9b77c27b2e67041fd2245") } 
> db.stuff.find() 
Fri Jan 18 13:29:28 Socket recv() errno:60 Operation timed out 192.168.1.111:12345 
Fri Jan 18 13:29:28 SocketException: remote: 192.168.1.111:12345 error: 9001 socket exception [1] server [192.168.1.111:12345] 
Fri Jan 18 13:29:28 DBClientCursor::init call() failed 
Fri Jan 18 13:29:28 query failed : mydatabase.stuff {} to: ds012345.mongolab.com:12345 
Error: error doing query: failed 
Fri Jan 18 13:29:28 trying reconnect to ds012345.mongolab.com:12345 
Fri Jan 18 13:29:28 reconnect ds012345.mongolab.com:12345 ok 
> db.stuff.find() 
{ "_id" : ObjectId("50f9b77c27b2e67041fd2245") }

We're here to help

Of course, if you have any questions please feel free to contact us at [email protected]. We're here to help.

Triton answered 18/1, 2013 at 23:42 Comment(5)
The support article to which you linked is no longer available.Bianco
Sorry about that. I've updated the post. You didn't miss anything, though. As you can see, the doc is essentially the contents of this post in a slightly different format.Triton
All right! Thank you! It's a shame the MaxConnectionIdleTime option is not available in the PyMongo driver (yet).Bianco
@Triton Is maxIdleTimeMS supported by node.js mongodb native driver?Libidinous
According to the driver docs there isn't a maxIdleTimeMS, but server.socketOptions.keepAlive might be what you're looking for.Triton
G
21

Thanks for all the help guys - I have managed to solve this issue on both localhost and deployed to a live server.

Here is my now working connect code:

var MONGO = {
    username: "username",
    password: "pa55W0rd!",
    server: '******.mongolab.com',
    port: '*****',
    db: 'dbname',
    connectionString: function(){
        return 'mongodb://'+this.username+':'+this.password+'@'+this.server+':'+this.port+'/'+this.db;
    },
    options: {
        server:{
            auto_reconnect: true,
            socketOptions:{
                connectTimeoutMS:3600000,
                keepAlive:3600000,
                socketTimeoutMS:3600000
            }
        }
    }
};

var db = mongoose.createConnection(MONGO.connectionString(), MONGO.options);

db.on('error', function(err) {
    console.log("DB connection Error: "+err);
});
db.on('open', function() {
    console.log("DB connected");
});
db.on('close', function(str) {
    console.log("DB disconnected: "+str);
});

I think the biggest change was to use "createConnection" over "connect" - I had used this before, but maybe the options help now. This article helped a lot http://journal.michaelahlers.org/2012/12/building-with-nodejs-persistence.html

If I'm honest I'm not overly sure on why I have added those options - as mentioned by @jareed, i also found some people having success with "MaxConnectionIdleTime" - but as far as i can see the javascript driver doesn't have this option: this was my attempt at trying to replicate the behavior.

So far so good - hope this helps someone.

UPDATE: 18 April 2013 note, this is a second app with a different setup

Now I thought i had this solved but the problem rose it's ugly head again on another app recently - with the same connection code. Confused!!!

However the set up was slightly different…

This new app was running on a windows box using IISNode. I didn't see this as significant initially.

I read there were possibly some issues with mongo on Azure (@jareed), so I moved the DB to AWS - still the problem persisted.

So i started playing about with that options object again, reading up quite a lot on it. Came to this conclusion:

options: {
    server:{
        auto_reconnect: true,
        poolSize: 10,
        socketOptions:{
            keepAlive: 1
        }
    },
    db: {
        numberOfRetries: 10,
        retryMiliSeconds: 1000
    }
}

That was a bit more educated that my original options object i state. However - it's still no good.

Now, for some reason i had to get off that windows box (something to do with a module not compiling on it) - it was easier to move than spend another week trying to get it to work.

So i moved my app to nodejitsu. Low and behold my connection stayed alive! Woo!

So…. what does this mean… I have no idea! What i do know is is those options seem to work on Nodejitsu…. for me.

I believe IISNode uses some kind of "forever" script for keeping the app alive. Now to be fair the app doesn't crash for this to kick in, but i think there must be some kind of "app cycle" that is refreshed constantly - this is how it can do continuous deployment (ftp code up, no need to restart app) - maybe this is a factor; but i'm just guessing now.

Of course all this means now, is this isn't solved. It's still not solved. It's just solved for me in my setup.

Gremlin answered 19/1, 2013 at 20:21 Comment(4)
Rather quirky, this is what did it for me: using mongoose.createConnection instead of connect Anyway, a multitude of thanks.Polyanthus
@Gremlin - the article you linked to seems to be missing it's examples. Do you know of an alternative copy? ThanksNarah
@Narah Sorry, i do not.Gremlin
I have used similar setup to this, due to my "zero downtime deploy" ( duplicate cluster then change DNS ) i end up with hundred of connections to mongodb, which completely slowed down my production environment. Now trying to purge the connections on SIGINT/SIGQUIT, trying to get rid of page faults... mongodb connection hellInadvertent
D
6

A couple of recommendations for people still having this issue:

  1. Make sure you are using the latest mongodb client for node.js. I noticed significant improvements in this area when migrating from v1.2.x to v1.3.10 (the latest as of today)

  2. You can pass an options object to the MongoClient.connect. The following options worked for me when connecting from Azure to MongoLab:

    options = { db: {}, server: { auto_reconnect: true, socketOptions: {keepAlive: 1} }, replSet: {}, mongos: {} };

    MongoClient.connect(dbUrl, options, function(err, dbConn) { // your code });

  3. See this other answer in which I describe how to handle the 'close' event which seems to be more reliable. https://mcmap.net/q/470952/-how-to-ensure-node-js-keeps-running-after-monogdb-connection-drops

Doucet answered 27/6, 2013 at 2:49 Comment(0)
E
1

Enable the auto_reconnect Server option like this:

var db = mongoose.connect(mongoConnect, {server: {auto_reconnect: true}});

The connection you're opening here is actually a pool of 5 connections (by default) so you're right to just connect and leave it open. My guess is that you intermittently lose connectivity with mongolab and your connections die when that occurs. Hopefully, enabling auto_reconnect resolves that.

Escalera answered 22/12, 2012 at 3:21 Comment(5)
okay cool. I'll give that a try and let you know. I thought auto reconnect was already enabled? And i don't get any disconnected events fired - I'll give it a go.Gremlin
No, sorry - the MongoDB has always been at mongoLab. But my code has been hosted locally, on Azure and on nodejit.su - maybe it's mongoLab, only I can't believe they would be able to offer a service if this was common. That's why I figured it must be me.Gremlin
Isn't auto_reconnect: true the default?Narah
@Narah Not according to the docs, no. Not sure why.Escalera
@Narah @Escalera auto_reconnect is true by default when using Mongoose: mongoosejs.com/docs/connections.htmlEthaethan
I
1

Increasing timeouts may help.

  • "socketTimeoutMS" : How long a send or receive on a socket can take before timing out.
  • "wTimeoutMS" : It controls how many milliseconds the server waits for the write concern to be satisfied.
  • "connectTimeoutMS" : How long a connection can take to be opened before timing out in milliseconds.

    $m = new MongoClient("mongodb://127.0.0.1:27017", array("connect"=>TRUE, "connectTimeoutMS"=>10, "socketTimeoutMS"=>10, "wTimeoutMS"=>10));

        $db= $m->mydb;
        $coll = $db->testData;
        $coll->insert($paramArr);
    
Inwardly answered 18/2, 2014 at 10:53 Comment(0)
T
0

I had a similar problem being disconnected from MongoDB periodically. Doing two things fixed it:

  1. Make sure your computer never sleeps (that'll kill your network connection).
  2. Bypass your router/firewall (or configure it properly, which I haven't figured out how to do yet).
Tramel answered 16/1, 2013 at 17:46 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.