I'm getting strange behavior in memcached, in particular, behavior that is strange in its consistency. Here is my test:
@Test
public void testMemc() {
logger.info("Setting head.");
memc.set(env.memcachedQueueKeys().head, 3600, 0);
logger.info("Set head; incrementing.");
memc.incr(env.memcachedQueueKeys().head, 1);
logger.info("Incremented.");
}
And here is the output:
28 11:04:52.932 INFO; Setting head.
2014-01-28 11:04:52.933 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for q:unittest:scannedemails:w.
28 11:04:52.933 INFO; Set head; incrementing.
2014-01-28 11:04:52.935 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for q:unittest:scannedemails:w.
FAILED: testMemc net.spy.memcached.OperationTimeoutException: Mutate operation timed out,unable to modify counter [q:unittest:scannedemails:w]
at net.spy.memcached.MemcachedClient.mutate(MemcachedClient.java:1484)
at net.spy.memcached.MemcachedClient.incr(MemcachedClient.java:1529)
at me.unroll.emailroller.ActOnScanResultsTest.testMemc(ActOnScanResultsTest.java:295)
Most of my intuition for this kind of error fails me here. The following things are all strange:
- Why does it always fail exactly once to set?
- Why does it permanently fail to increment after seeming to succeed at set?
This is on a high-load server (yes, it's a little wrong to be running a test on a load-bearing server, but if it catches issues like this there's at least some advantage). What can cause this consistent failure? There is only one node.