MongoConnectionException - No candidate servers found

Question

I am developing a PHP web application using a MongoDB replicaset to store my data. I occasionally receive the following error:

Fatal error: Uncaught exception 'MongoConnectionException' with message 'No candidate servers found'

I have a 3 member Mongo replica set with 1 arbiter

rs0:PRIMARY> rs.status()
{
    "set" : "rs0",
    "date" : ISODate("2013-01-30T01:04:04Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "JenEricsMacPro.local:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 844478,
            "optime" : Timestamp(1359507378000, 1),
            "optimeDate" : ISODate("2013-01-30T00:56:18Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "ericsmacbookpro.local:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 10720,
            "optime" : Timestamp(1359507378000, 1),
            "optimeDate" : ISODate("2013-01-30T00:56:18Z"),
            "lastHeartbeat" : ISODate("2013-01-30T01:04:04Z"),
            "pingMs" : 3
        },
        {
            "_id" : 2,
            "name" : "ericsmacbookair.local:27017",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 1206,
            "lastHeartbeat" : ISODate("2013-01-30T01:04:03Z"),
            "pingMs" : 4
        },
        {
            "_id" : 3,
            "name" : "ericsmacxps.local:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 75204,
            "optime" : Timestamp(1359507378000, 1),
            "optimeDate" : ISODate("2013-01-30T00:56:18Z"),
            "lastHeartbeat" : ISODate("2013-01-30T01:04:03Z"),
            "pingMs" : 1
        }
    ],
    "ok" : 1
}

Below is an example of the code where I occasionally receive this error.

$mongoConfig = array (
    'uri' => 'mongodb://JenEricsMacPro.local:27017,ericsmacbookpro.local:27017,ericsmacxps.local:27017/',
    'database' => 'myDatabase',
    'replicaSetArray' => array('replicaSet' => 'rs0'));

$connection = new MongoClient(
    $mongoConfig['uri'],
    $mongoConfig['replicaSetArray']);

$connection->setReadPreference(MongoClient::RP_SECONDARY_PREFERRED);

$db = $connection->selectDB($mongoConfig['database']);

Anyone have any idea what might be causing the 'No candidate servers found' message or any suggestions on how I could determine the root of the problem?

All of my mongo.conf files are using the static IP of each server, no firewalls and DNS appears to be resolving correctly to the .local domain on my network.

EDIT: I added logging, suggested by Derick, prior to instantiating MongoClient.

error_reporting (E_NOTICE);

MongoLog::setModule( MongoLog::ALL );
MongoLog::setLevel( MongoLog::ALL );

After enabling logging I received the following notices.

Notice: PARSE INFO: Parsing mongodb://JenEricsMacPro.local:27017,ericsmacbookpro.local:27017,ericsmacxps.local:27017/ in MongoLogic.php on line 27
Notice: PARSE INFO: - Found node: JenEricsMacPro.local:27017 in MongoLogic.php on line 27
Notice: PARSE INFO: - Found node: ericsmacbookpro.local:27017 in MongoLogic.php on line 27
Notice: PARSE INFO: - Found node: ericsmacxps.local:27017 in MongoLogic.php on line 27
Notice: PARSE INFO: - Connection type: MULTIPLE in MongoLogic.php on line 27
Notice: PARSE INFO: - Found option 'replicaSet': 'rs0' in MongoLogic.php on line 27
Notice: PARSE INFO: - Switching connection type: REPLSET in MongoLogic.php on line 27
Notice: CON INFO: mongo_get_read_write_connection: finding a REPLSET connection (read) in MongoLogic.php on line 27
Notice: CON FINE: found connection JenEricsMacPro.local:27017;rs0;X;297 (looking for JenEricsMacPro.local:27017;rs0;X;297) in MongoLogic.php on line 27
Notice: CON INFO: is_ping: pinging JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: mongo_connection_destroy: Closing socket for JenEricsMacPro.local:27017;rs0;X;297. in MongoLogic.php on line 27
Notice: CON INFO: freeing connection JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: Couldn't connect to 'JenEricsMacPro.local:27017': send_package: error reading from socket: Operation timed out in MongoLogic.php on line 27
Notice: CON FINE: found connection ericsmacbookpro.local:27017;rs0;X;297 (looking for ericsmacbookpro.local:27017;rs0;X;297) in MongoLogic.php on line 27
Notice: CON INFO: is_ping: pinging ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: mongo_connection_destroy: Closing socket for ericsmacbookpro.local:27017;rs0;X;297. in MongoLogic.php on line 27
Notice: CON INFO: freeing connection ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: Couldn't connect to 'ericsmacbookpro.local:27017': send_package: error reading from socket: Operation timed out in MongoLogic.php on line 27
Notice: CON FINE: found connection ericsmacxps.local:27017;rs0;X;297 (looking for ericsmacxps.local:27017;rs0;X;297) in MongoLogic.php on line 27
Notice: CON INFO: is_ping: pinging ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: mongo_connection_destroy: Closing socket for ericsmacxps.local:27017;rs0;X;297. in MongoLogic.php on line 27
Notice: CON INFO: freeing connection ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: Couldn't connect to 'ericsmacxps.local:27017': send_package: error reading from socket: Operation timed out in MongoLogic.php on line 27
Notice: CON FINE: discover_topology: checking ismaster for JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: discover_topology: couldn't create a connection for JenEricsMacPro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: discover_topology: checking ismaster for ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: discover_topology: couldn't create a connection for ericsmacbookpro.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON FINE: discover_topology: checking ismaster for ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: CON WARN: discover_topology: couldn't create a connection for ericsmacxps.local:27017;rs0;X;297 in MongoLogic.php on line 27
Notice: REPLSET FINE: finding candidate servers in MongoLogic.php on line 27
Notice: REPLSET FINE: - all servers in MongoLogic.php on line 27
Notice: REPLSET FINE: filter_connections: adding connections: in MongoLogic.php on line 27
Notice: REPLSET FINE: filter_connections: done in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting to servers with same replicaset name in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting to servers with same replicaset name: done in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting by credentials in MongoLogic.php on line 27
Notice: REPLSET FINE: limiting by credentials: done in MongoLogic.php on line 27

Damselfish

Do the hostnames provided in your mongo config actually resolve from the computer you are running this php script on? If you did ping JenEricsMacPro.local would it resolve to an ip address? – Sweepstake

Yes, they resolve correctly using ping and nslookup. – Damselfish

There seems to be related issues such as: groups.google.com/forum/?fromgroups=#!topic/mongodb-user/… Try googling the error message for more. – Dupondius

Thanks. I had previously run across same article. I removed the bind_id = localhost from the config files on each server and went with 10.0.0.x local IPs. For example, one server's mongo.conf contains bind_id = 10.0.0.50 now, however I still receive the error. Each server in my replica set is a Mac. At one point, Windows and Linux machines couldn't find my macservernames.local on the network so I added all 4 of the .local addresses to my LAN's DNS server figuring that might be related. That didn't help. I'm really scratching my head on this one. The problem comes and goes. – Damselfish

Answer 1

I would suggest you add logging to find out what is happening. The Mongo driver (since version 1.3) provides extensive logging to debug connection issues. There is an article at http://derickrethans.nl/mongodb-debugging.html to explain how to turn it on. If from the log info it is not clear what happens, update your question with the log, so that I can update my answer.

From the posted log I can see that your client can not connect to any of the seeds from your connection string. If your machine/client can't connect, you can't talk to MongoDB of course. In this case, there can be a few possibilities:

• The driver can't convert the names to IP addresses. In that case, you need to fix your DNS setup, or add all hosts to /etc/hosts
• You really can't connect to them, this could be a firewall in the way.
• There are some issues with that in the 1.2 series of the driver, so if you're not using 1.3.4 or higher yet, I'd suggest you upgrade. It should be a much smoother experience.
• Remember that you need a majority of your nodes up in order to be able to write. If you don't have that, then you will get the "No candidate servers" error.

Aphis

Turned on driver logging and updated my question. I see errors reading from socket. – Damselfish

Not 100% sure this is related. I remove EricsMacBookAir and EricsMacBookPro from the network everyday. When I come back to the network where the remaining replica set exists and start developing in php I hit the error. I saw another post where the php mongo driver sometimes persists old connections. When I returned to the replica set today I immediately restarted Apache prior to coding and the error hasn't occurred. Related link: #14250268 – Damselfish

@geekinit, there are some issues with that in the 1.2 series of the driver, so if you're not using 1.3.4 or higher yet, I'd suggest you upgrade. It should be a much smoother experience. Do however remember that you need a majority of your nodes up in order to be able to write. If you don't have that, then you will get the "No candidate servers" error.—I've updated the answer with this. – Aphis

I updated to version 1.3.4 and it appears the problem may be corrected now. Thanks so much for you help! – Damselfish

Updating mongo driver to 1.3.7 helps me! – Sanborn