Connect to MongoDB Replica Set running inside Docker with Java (Windows)

Question

I want to set up a MongoDB replica set with docker. The setup seems to be fine but I'm not able to connect to the cluster with my Java-app. I'm running Docker version 17.06.0-ce on Windows 10 with VirtualBox as driver.

I followed the instructions from this tutorial: http://www.sohamkamani.com/blog/2016/06/30/docker-mongo-replica-set/

So I first created the my-mongo-cluster network in docker and run 3 containers with the following commands:

$ docker run --name mongo1 -d --net mongo-cluster -p 9042:27017 mongo:3.6.0 mongod --replSet my-mongo-set
$ docker run --name mongo2 -d --net mongo-cluster -p 9142:27017 mongo:3.6.0 mongod --replSet my-mongo-set
$ docker run --name mongo3 -d --net mongo-cluster -p 9242:27017 mongo:3.6.0 mongod --replSet my-mongo-set

Then I connect to the mongo1 container and set up the replica set with the following config:

config = {"_id" : "my-mongo-set", "members" : [{"_id" : 0,"host" : "mongo1:27017"},{"_id" : 1,"host" : "mongo2:27017"},{"_id" : 2,"host" : "mongo3:27017"}]}
rs.initiate(config)

This seems to work just fine. As I can tell from the logs of the mongo1 container all containers are connected to each other.

Now I'm trying to connect from my Java-app to the replica set. I'm using the mongodb-driver version 3.6.0. This is the code I'm using to connect to the replica set running in docker:

List<ServerAddress> serverAddresses = new ArrayList<ServerAddress>();
serverAddresses.add(new ServerAddress(InetAddress.getByName("192.168.99.100"), 9042));
serverAddresses.add(new ServerAddress(InetAddress.getByName("192.168.99.100"), 9142));
serverAddresses.add(new ServerAddress(InetAddress.getByName("192.168.99.100"), 9242));

MongoClient client = new MongoClient(serverAddresses);

And this is the log-output (the MongoSocketException at the end also appears for mongo1:27017 and mongo2:27017):

Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Cluster created with settings {hosts=[192.168.99.100:9042, 192.168.99.100:9142, 192.168.99.100:9242], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Adding discovered server 192.168.99.100:9042 to client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Adding discovered server 192.168.99.100:9142 to client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Adding discovered server 192.168.99.100:9242 to client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Cluster description not yet available. Waiting for 30000 ms before timing out
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Opened connection [connectionId{localValue:2, serverValue:5}] to 192.168.99.100:9142
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Opened connection [connectionId{localValue:1, serverValue:14}] to 192.168.99.100:9042
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Opened connection [connectionId{localValue:3, serverValue:5}] to 192.168.99.100:9242
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Monitor thread successfully connected to server with description ServerDescription{address=192.168.99.100:9242, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5002651, setName='my-mongo-set', canonicalAddress=mongo3:27017, hosts=[mongo3:27017, mongo2:27017, mongo1:27017], passives=[], arbiters=[], primary='mongo1:27017', tagSet=TagSet{[]}, electionId=null, setVersion=1, lastWriteDate=Wed Dec 27 21:36:00 CET 2017, lastUpdateTimeNanos=440549516217673}
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Monitor thread successfully connected to server with description ServerDescription{address=192.168.99.100:9142, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5542139, setName='my-mongo-set', canonicalAddress=mongo2:27017, hosts=[mongo3:27017, mongo2:27017, mongo1:27017], passives=[], arbiters=[], primary='mongo1:27017', tagSet=TagSet{[]}, electionId=null, setVersion=1, lastWriteDate=Wed Dec 27 21:36:00 CET 2017, lastUpdateTimeNanos=440549516254709}
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Monitor thread successfully connected to server with description ServerDescription{address=192.168.99.100:9042, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4113767, setName='my-mongo-set', canonicalAddress=mongo1:27017, hosts=[mongo3:27017, mongo2:27017, mongo1:27017], passives=[], arbiters=[], primary='mongo1:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=1, lastWriteDate=Wed Dec 27 21:36:00 CET 2017, lastUpdateTimeNanos=440549515190458}
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Discovered cluster type of REPLICA_SET
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Adding discovered server mongo3:27017 to client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Adding discovered server mongo2:27017 to client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Adding discovered server mongo1:27017 to client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Canonical address mongo2:27017 does not match server address.  Removing 192.168.99.100:9142 from client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Server 192.168.99.100:9242 is no longer a member of the replica set.  Removing from client view of cluster.
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Server 192.168.99.100:9042 is no longer a member of the replica set.  Removing from client view of cluster.
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Canonical address mongo1:27017 does not match server address.  Removing 192.168.99.100:9042 from client view of cluster
Dez 27, 2017 9:36:07 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: No server chosen by com.mongodb.Mongo$4@67784306 from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=192.168.99.100:9242, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo3:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo2:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo1:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=192.168.99.100:9042, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
Dez 27, 2017 9:36:09 PM com.mongodb.diagnostics.logging.JULLogger log
INFORMATION: Exception in monitor thread while connecting to server mongo3:27017
com.mongodb.MongoSocketException: mongo3
at com.mongodb.ServerAddress.getSocketAddress(ServerAddress.java:188)
at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:59)
at com.mongodb.connection.SocketStream.open(SocketStream.java:57)
at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:126)
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:114)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: mongo3
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getByName(InetAddress.java:1076)
at com.mongodb.ServerAddress.getSocketAddress(ServerAddress.java:186)
... 5 more

Canty

isn't it easier to use docker-compose to run those three replicas in a simple network? Also, not quite sure, but in your config json the host shouldn't be pointing to the 9042... ports? – Congratulate

Answer 1

Finally I found the solution by reading this:

https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#clients-use-the-hostnames-listed-in-the-replica-set-config-not-the-seed-list

The last sentence is important: "In conclusion, to support key features of replica sets, we require that the hostnames used in a replica set config are reachable from the client."

So this means that the hostnames in the config have to be visible/accessible from outside. But since I'm running Docker on Windows with VirtualBox the MongoClient only can see the IP of my VM and can't handle mongo1, mongo2, mongo3 (respectively the internal IPs of the containers).

The solution was to add the hostnames to the hosts-file (in Windows located at C:/Windows/System32/drivers/etc/hosts):

192.168.99.100 mongo1 192.168.99.100 mongo2 192.168.99.100 mongo3

Canty

Is there another solution when you don't have admin right to modify etc/host file ??? – Piers

Answer 2

Adding the hostnames to the hosts file did not work for me. I think if all hostnames refers to the same host IP (e.g. 127.0.0.1), it's not going to work if all docker ports are the same (e.g. 27017). The replica set is composed by mongo1:27017, mongo2:27017 and mongo3:27017 inside docker. Outside docker it corresponds to 127.0.0.1:27017, 127.0.0.1:27017 and 127.0.0.1:27017 which won't work. To fix the issue I had to set a different port for each node.

docker network create mongo-cluster
docker run --name mongo1 -d --net mongo-cluster -p 9042:9042 mongo:3.6 mongod --replSet docker-rs --port 9042
docker run --name mongo2 -d --net mongo-cluster -p 9142:9142 mongo:3.6 mongod --replSet docker-rs --port 9142
docker run --name mongo3 -d --net mongo-cluster -p 9242:9242 mongo:3.6 mongod --replSet docker-rs --port 9242
docker exec -it mongo1 mongo --port 9042
config = {"_id" : "docker-rs", "members" : [{"_id" : 0,"host" : "mongo1:9042"},{"_id" : 1,"host" : "mongo2:9142"},{"_id" : 2,"host" : "mongo3:9242"}]}
rs.initiate(config)
rs.status() 

and finally add the hostnames to the hosts file

127.0.0.1 mongo1 mongo2 mongo3

Wesleywesleyan

You literally saved my life. Thank you! – Kenweigh

Saved me too! I guess --port did the thing. Am I right? @Wesleywesleyan @Marek M. – Carlo

Is there another solution when you don't have admin right to modify etc/host file ??? – Piers

Answer 3

Problem is that your replica set nodes are seen outside at ports 9x42 but those nodes internal (configured) addresses are 27017. Both those must be same.So, when you start mongod you must give --port parameter with same port number what is used at outside of container.

Depolymerize

Thanks. I still can't get it work. I tried starting the containers: docker run --name mongo1 ... -p 27017:27017 mongo mongod --replSet my-mongo-set --port 27017 and ` docker run --name mongo2 ... -p 27018:27018 mongo mongod --replSet my-mongo-set --port 27018 and the third one with port 27019. Also I changed the config to mongo1:27017, mongo2:27018 and mongo3:27019 and in my Java-app I'm building the server addresses with this 3 ports. But still the same exception... "Canonical address mongo2:27018 does not match server address. Removing 192.168.99.100:27018 from client view of cluster" – Canty

You need to add lines to /etc/hosts files at every node, so that nodes name like mongo2 is resolved to that IP address what it should be. As you can see from error message, it says that mongo2 is not resolving to that 192.168.99.102 – Depolymerize

I've already found the solution and answered the question on my own. Thank you for your help :) – Canty