Mongo CursorNotFound exception in active cursor via Grails domain criteria

I'm using Grails 2.4.4, mongo plugin 3.0.2, MongoDB 2.4.10 using a remote database connection.

grails {
    mongo {
        host = "11.12.13.14"    // A remote server IP
        port = 27017
        databaseName = "blogger"
        username = "blog"
        password = "xyz"
        options {
            autoConnectRetry = true
            connectTimeout = 3000
            connectionsPerHost = 40
            socketTimeout = 120000
            threadsAllowedToBlockForConnectionMultiplier = 5
            maxAutoConnectRetryTime=5
            maxWaitTime=120000
        }
    }
}

In a part of our application, a service method iterates over a 20,000 user's and sends them an email:

Person.withCriteria {       // Line 323
    eq("active", true)
    order("dateJoined", "asc")
}.each { personInstance ->
   // Code to send an email which takes an average of 1 second
}

After executing this for some 6000 user, I'm getting a MongoDB cursor exception:

2015-04-11 07:31:14,218 [quartzScheduler_Worker-1] ERROR listeners.ExceptionPrinterJobListener  - Exception occurred in job: Grails Job
org.quartz.JobExecutionException: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017 [See nested exception: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017]
        at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:111)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017
        at com.mongodb.QueryResultIterator.throwOnQueryFailure(QueryResultIterator.java:218)
        at com.mongodb.QueryResultIterator.init(QueryResultIterator.java:198)
        at com.mongodb.QueryResultIterator.initFromQueryResponse(QueryResultIterator.java:176)
        at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:141)
        at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
        at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
        at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
        at org.grails.datastore.mapping.mongo.query.MongoQuery$MongoResultList$1.hasNext(MongoQuery.java:1893)
        at com.test.person.PersonService.sendWeeklyEmail(PersonService.groovy:323)
        at com.test.WeeklyJob.execute(WeeklyJob.groovy:41)
        at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104)
        ... 2 more

I looked for documentation and found that cursor automatically get closed in 20 minutes and when I confirmed it with the logs, this exception came exactly after 20 minutes.

But this behaviour of auto-close in 20 minutes is applicable for inactive cursor but here the cursor is active.

UPDATE:

I read some articles and found that, this could be an issue of TCP keepalive timeout. So we changed the TCP keepalive timeout to 2 minutes from default 2 hours, but it still doesn't solve the problem.

Recommended topics

Hot tags