MongoException: java.lang.OutOfMemoryError: GC overhead limit exceeded

Question

I am bulk writing to MongoDB and get an OOM exception (java.lang.OutOfMemoryError: GC overhead limit exceeded). I have two questions:

1. Does the OOM come from the Mongo Client Driver or MongoDB Server?
2. Is there a clue why the OOM happens?

FO 2016-11-15 15:19:10,437 - [TS] org.mongodb.driver.cluster info(71) - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, all=[ServerDescription{address=mongo.server1-or:30000, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo.server2:30000, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo.server3:30000, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out INFO 2016-11-15 15:19:11,448 - [TS] org.mongodb.driver.cluster info(71) - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, all=[ServerDescription{address=mongo.server1-or:30000, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo.server2:30000, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo.server3:30000, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out INFO 2016-11-15 15:19:14,324 - [TS] org.mongodb.driver.cluster info(76) - Exception in monitor thread while connecting to server mongo.server2:30000 com.mongodb.MongoException: java.lang.OutOfMemoryError: GC overhead limit exceeded at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:125) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:128) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded INFO 2016-11-15 15:19:14,325 - [TS] com.xyz.executors.ConsumeMessageTask run(45) - Scheduled thread pool will write to MongoDB INFO 2016-11-15 15:19:14,325 - [TS] org.mongodb.driver.connection info(71) - Opened connection [connectionId{localValue:28690}] to mongo.server3:30000 ERROR 2016-11-15 15:19:17,353 - [TS] com.xyz.executors.ConsumeMessageTask run(117) - An error occurred when bulk writing to MongoDB com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongo.server1-or:30000, type=UNKNOWN, state=CONNECTING}, {address=mongo.server2:30000, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoException: java.lang.OutOfMemoryError: GC overhead limit exceeded}, caused by {java.lang.OutOfMemoryError: GC overhead limit exceeded}}, {address=mongo.server3:30000, type=UNKNOWN, state=CONNECTING}] at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369) at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101) at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75) at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71) at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219) at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168) at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74) at com.mongodb.Mongo.execute(Mongo.java:781) at com.mongodb.Mongo$2.execute(Mongo.java:764) at com.mongodb.DBCollection.executeBulkWriteOperation(DBCollection.java:2195) at com.mongodb.DBCollection.executeBulkWriteOperation(DBCollection.java:2188) at com.mongodb.BulkWriteOperation.execute(BulkWriteOperation.java:121) at com.xyz.executors.ConsumeMessageTask.run(ConsumeMessageTask.java:112) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) `

Vannesavanness

Answer 1

Your stacktrace says that the error originates from this line:

at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369) at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)...

so the client times out when trying to connect to the server. The information that was received from the server is on the line above:

Client view of cluster state is {type=UNKNOWN, servers=[ {address=mongo.server1-or:30000, type=UNKNOWN, state=CONNECTING}, {address=mongo.server2:30000, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoException: java.lang.OutOfMemoryError: GC overhead limit exceeded}, caused by {java.lang.OutOfMemoryError: GC overhead limit exceeded}}, {address=mongo.server3:30000, type=UNKNOWN, state=CONNECTING}]

This means that the OutOfMemoryError is coming from mongo.server2. It's best the check the logs of that server to see why it occurs, but the bulk writing could have something to do with it. It could be that mongo.server2 can't keep up with syncing the data you write to the primary server.

Fermin