Couchbase bad connectivity (TIMING OUT) in AWS VPC

Question

I am experiencing a connectivity problem which is a bizarre in a private virtual vpc in Amazon environment (AWS). My context is like this i have subnet in in this subnet i have 2 machines a client machine where i used Couchbase java SDK and a machine for Couchbase server. Both are large instances. Current java version 1.6 . I have open all the required ports (8091,8092,11210) for Couchbase on the Security Groups. I am testing the connection with a curl command to test the connectivity with a bucket and no problems here. The strange behavior is when i am trying to access a view with the java client because i receive a timeout error although the java client is authenticated successfully.

This is what my log look like:

(CacheManager.java:102) -B06C9F5CFF85- Cache client checked out [stdout]
INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.0.X.XXX:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@57a220c2
INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.0.X.XXX/10.0.X.XXX:11210
Added 10.0.X.XXX to connect queue
INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /10.0.X.XXX:8092 - Session request successful
ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [10.0.X.XXX/10.0.X.XXX:8092]
and then after a while
INFO com.couchbase.client.ViewConnection: Node 10.0.X.XXX has no ops in the queue
INFO com.couchbase.client.ViewNode: I/O reactor terminated for 10.0.X.XXX
SHUTTING DOWN (informed client)) INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client

So suddenly the connection times out without any reason and then the client it is shutting down itself.

Other context data: Couchbase server enterprise 2.0 Java 1.6 Ubuntu 11.10 server x64

On my local machine i have a Couchbase running in a local vm in local lan and i have no problems in using it.

Can anybody help in debugging this. We want to take Couchbase to production but without this we can not advance.

Thank you in advance for any help or information.

Taproom

I'm using amazon too, but ec2 instances. Sometimes they loosing connection between each other. Try to run some script like ping in cron (or use some monitoring tool like munin, etc.) to determine what cause connection down. May be it's amazon issue. – Quincuncial

Answer 1

Per couchbase client documentation -

The default timeout for any given node in a Couchbase cluster is 2.5 seconds. If a Couchbase SDK does not receive a response from the server by this time, it will drop the connection to Couchbase Server and attempt to connect to another node.

You may want to change the "OpTimeout" value in case your connectivity is not good or unreliable.

CouchbaseConnectionFactoryBuilder b = new CouchbaseConnectionFactoryBuilder();
            b.setOpTimeout(5000);
            client = new CouchbaseClient(b.buildCouchbaseConnection(nodes, "bucket-name", "pass"));

Overabundance

this is exactly what I was looking for, thanks! just one remark: the CouchbaseClient does not have this constructor (anymore?) so it has to be done this way: CouchbaseConnectionFactory ccf = b.buildCouchbaseConnection(nodes, "bucket-name", "pass"); CouchbaseClient client = new CouchbaseClient(ccf); – Vary