node-mongodb-native MongoClient unexpectedly closing connections

Question

I've been searching a lot for unexpectedly closed connections in mongodb but can only find questions from people who WANT their connections to close.

I am using node-mongodb-native to connect to a db, but I keep getting seemingly random "Error: connection closed" messages. If I manually retry the request (browser refresh) the request works.

Any idea what is causing this? Is there some simple option that will help?

I'm getting my db handle using:

     MongoClient.connect(connection_string, { auto_reconnect: true }, function (err, db) {
     //server code/routes in here
     }

I was looking through https://github.com/mongodb/node-mongodb-native/blob/master/lib/mongodb/connection/server.js but I realize my limited understanding of how connection pools are managed in general is tripping me up. I was under the impression they would stay open for the lifetime of my server. Can someone help?

Edit: After reading mjhm's comments, I started looking more deeply into TCP keep alive. Stumbled across some sites that suggest this may be Azure's doing (and this question is now misclassified!). Apparently, the Azure load balancer kills connections after 1 min of activity. I'm using Azure Websites, so it may or may not apply, but I think this insight is promising enough to start a new line of investigation. More details here http://blogs.msdn.com/b/avkashchauhan/archive/2011/11/12/windows-azure-load-balancer-timeout-details.aspx

Edulcorate

Can you provide more information? Is your DB localhost or another machine? What db calls are triggering the connection closed messages? How much load is on the server? Etc. – Mesotron

Hi, the db is hosted on mongohq. There is very little load on the service. I cant reliably reproduce the issue, but I notice the find() calls are failing for sure. I tried hardcoding the keepAlive flag to true in the node-mongodb-native drive, but that didnt work either. – Edulcorate

This note has some suggestions, christiankvalheim.com/post/32209721702/tcp-keepalive – Mesotron

Thanks! I think you are right. I can mark that as an answer. – Edulcorate

Answer 1

From: http://christiankvalheim.com/post/32209721702/tcp-keepalive

TCP keepalive One thing that comes up quite frequently as a question when using the mongodb node.js driver is a socket that stops responding. This usually have two sources.

There is a firewall in between the application and the mongodb instance and it does not observe keepAlive.

The socket timeout is to high on your system leaving the socket hanging and never closing. The first situation can be remedied by setting the socket connection options and enabling keepAlive and setting a hard timeout value on the socket. This will ensure that a correctly configured firewall will keep the connection alive and if it does not it will timeout. The other thing to tweak is the os tcp_keepalive_time. Basically it’s to high for something like MongoDB (default 2 hours on linux). Setting this lower will correctly timeout dead sockets and let the driver recover.

A good link to read more about it. http://www.mongodb.org/display/DOCS/Troubleshooting#Troubleshooting-Socketerrorsinshardedclustersandreplicasets

Mesotron