How to handle recovery memcached nodes when using spymemcached & HashAlgorithm.KETAMA_HASH

Question

I am using spymemcached & HashAlgorithm.KETAMA_HASH to connect to a pool of memcached of 5 nodes.

My understanding is when we use a consistent hashing algorithm like, when a node is down, we don't need to worry as the key will be re-distributed (with min. impact)

What if when the down-ed node is going to join the pool. What I need to do?

Should I make sure stale data need to be removed? Or should my program need special handling for this case?

Mcpeak

Answer 1

Given that this document is accurate: http://info.couchbase.com/rs/northscale/images/Couchbase_WP_Dealing_with_Memcached_Challenges.pdf

If there is any network disruption, and one or more clients decide that a particular memcached server is not available anymore, they will automatically rehash some data into the rest of the nodes even if the original one is still available. If the node eventually returns to service (for example after the network outage is resolved), the data on that node will be out of date and the clients without updated key-server remapping info will read stale data.

Assuming this is still up to date: http://lists.danga.com/pipermail/memcached/2007-April/003852.html It would be safe refresh/flush the node before adding it back. Forcing the down-ed node to clear up any stale entry.

Grote

My understanding is I have to flush all stale data in all nodes. Why? Because some clients might seeing the down-ed node is up but some clients might still assume the node is down. So given that clients might hash the key into different node, data inconsistency might occur. But this problem is hard unless we have a way to lock all memcached nodes and perform the flush - to avoid race condition. – Mcpeak

Based on my opinion, I think it's still safe to only flush the down-ed node. It's clean, if there is any request to the node, it will result to a miss, adding new and fresh information. If the client think it is down, it will hash to another node, also resulting to a miss. If that happens, both node have the key 'foo' for instance. When the down-ed node comes back online to that client, you need to flush it again. Here is some more about this: caiapps.com/duplicate-key-problem-in-memcache-php – Loudmouthed

yes but what you said only valid when you have a singe client but multiple servers. there is no way to coordinate different clients to use the same set of memcached servers at a given time (assume using consistent hashing). – Mcpeak