Spymemcached and Connection Failures

Question

I though Spymemcached does attempt to reestablish connection to the server when this connection get lost.

But I am seeing something different; wondering what I misunderstand or what I do wrong. Here is some sample code:

 MemcachedClient c=new MemcachedClient(AddrUtil.getAddresses("server:11211"));
while(true)
try {
    System.out.println(c.get("hatsts"));
    Thread.sleep(10000);
} catch(Exception e) {
    e.printStackTrace();
}

It runs initially without problem. Then I pull the network plug. Subsequently, the client detects a network failure and throws following exception:

net.spy.memcached.OperationTimeoutException: Timeout waiting for value

But then, when i re-establish the network, the client does not recover and continues throwing the exception; even after 5 min. I tried SpyMemcached 2.10.6 and 2.9.0.

What am I missing?

Indignity

Answer 1

The problem here is that because you pulled the network cable the tcp socket on you client still thinks the connection is valid. The tcp keepalive varies from operating system to operating system and can be as high as 30 minutes. As a result the application (in this case Spymemcached) is not notified that the connection is no longer valid by the tcp layer and therefore Spymemcached doesn't try to reconnect.

The way Spymemcached detects this situation is by counting the amount of consecutive operation timeouts. The last time I checked the default value was 99. Once this many ops time out then Spymemcached will reconnect. You can change this value in the ConnectionFactory if you want to set it to some other value. There's a function called getContinuousTimeout() which is where the Spymemcached gets 99 from by default. You can construct your own ConnectionFactory with the ConnectionFactoryBuilder.

Hopefully this is enough information to answer your question and get you going in the right direction. If not let me know and I can add some more details.

Guinea

Indeed. Using net.spy.memcached.spring.MemcachedClientFactoryBean, the property that has to be tuned is called timeoutExceptionThreshold. I set it to 10 and I saw reconnection to Memcached happening nicely. – Indignity