Handling MongoDB disconnect/reconnects from Node

Question

When my MongoDB connection is idle for a few minutes, the next request ends in error. From the mongo command line client, it looks like this:

> db.users.find()
Sat Jan 12 23:42:35 Socket recv() errno:54 Connection reset by peer 107.22.25.25:47207
Sat Jan 12 23:42:35 SocketException: remote: 107.22.25.25:47207 error: 9001 socket exception [1] server [107.22.25.25:47207] 
Sat Jan 12 23:42:35 DBClientCursor::init call() failed
Sat Jan 12 23:42:35 query failed : chowology.users {} to: ds047207.mongolab.com:47207
Error: error doing query: failed
Sat Jan 12 23:42:35 trying reconnect to ds047207.mongolab.com:47207
Sat Jan 12 23:42:35 reconnect ds047207.mongolab.com:47207 ok

I see the problem against sandbox instances of MongoHQ and MongoLab.

The next request goes through fine, because of the reconnect. This is a problem in my web app, because after a few minutes of inactivity, this error will come up during a web request. There are two things that surprise me:

1. That MongoDB connections are destroyed so regularly and frequently, and
2. That the driver simply raises an exception as opposed to automatically retrying after reconnecting (I'm using connect-mongo which uses mongoose which, in turn, uses node-mongodb-native).

Is this everyone else's experience? How is this supposed to be handled? I'd be surprised if app developers wrap their database operations in some retry-exception-handling nonsense.

Farflung

This might be useful: #14159634 – Nonpareil

Answer 1

You want to look at the docs for the Server object

http://mongodb.github.com/node-mongodb-native/api-generated/server.html#server

Especially the socketOptions where you can set keepAlive and the connection timeouts. By default keepalive is off and timeout is 0 or never which means the os default socket timeout is in effect (varies from os to os). Keep alive will send a packet once in awhile down the tcp socket connection to keep it alive. Sometimes firewalls are badly configured and don't send an end packet when they close a connection leaving the connection dead and in limbo which is what the monoglabs people are talking about (more often than not to be honest they are horribly configured).

Bethel

Thanks for the details. After bypassing my router I no longer have the problem. Is this router/firewall configuration something I can do myself? I know nothing about configuring my router other than setting a password and opening incoming ports. – Farflung

completely depends on your router. Sorry I cannot be more helpful but router configuration is a especially horrible world of pain and each one is different. – Bethel

Update: Changing routers fixed this problem completely. Thanks again. – Farflung

Answer 2

1. Check that your computer isn't going to sleep
2. Check that your router/firewall is not killing idle connections

The first problem turned out to be my computer sleeping and dropping the network connection unknowingly. It's a new computer and I didn't realize I hadn't disabled the sleep :-P

Jared from MongoLab helped me troubleshoot this and I'm thankful for it. He said this behavior is common when going through a firewall (as mjhm suggested in his comment). So one test would be to bypass that.

Still going through my router, I get a different error after several hours idle:

db.users.find()
Sun Jan 13 14:55:02 Socket say send() errno:32 Broken pipe 107.22.25.25:47207
Error: 9001 socket exception [2] server [107.22.25.25:47207] 
Sun Jan 13 14:55:02 trying reconnect to ds047207.mongolab.com:47207
Sun Jan 13 14:55:02 reconnect ds047207.mongolab.com:47207 ok

I'll try it again from a server that doesn't go through my router/firewall.

The behavior of the driver raising the exception on the current op is expected and acceptable since a dropped connections is truly an exceptional case.

Update: Neither of these problems occur when I bypass my router, nor do they occur in my Nodejitsu instance, which I believe runs in a Joyent data center.

Farflung

Answer 3

I had the same problem and I think it's because I'm accessing the internet behind a proxy server

Tegument