memcache and wildcards

Question

I'm just wondering if there is a way to clear memcache using wildcards for key values.

So say I have a cache with the key "1234~foo" and another "1234~foo~bar".

Is there any way I can say clear the cache by using something like clear("1234*") and have it clear both from above?

I hope that makes sense.

Thanks.

Lydgate

Answer 1

No, there isn't a direct easy way to do this. The FAQ addresses this, and provides a kind of workaround:

Deleting by Namespace

While memcached does not support any type of wildcard deleting or deletion by namespace (since there are not namespaces), there are some tricks that can be used to simulate this. They do require extra trips to the memcached servers however.

Example, in PHP, for using a namespace called foo:

$ns_key = $memcache->get("foo_namespace_key");
// if not set, initialize it
if($ns_key===false) {
    $ns_key=rand(1, 10000);
    $memcache->set("foo_namespace_key", $ns_key);
}
// cleverly use the ns_key
$my_key = "foo_".$ns_key."_12345";
$my_val = $memcache->get($my_key);

//To clear the namespace do:
$memcache->increment("foo_namespace_key");

Lamere

This solution is so cool. But why on earth would memcached planners overlook such a (seemingly) integral features as deleting by wildcard? I'm guessing some sort of performance reason. Does anyone know? – Urger

They didn't overlook it -- rather, the data structure isn't built to handle it. Memcache's big thing is raw speed, and there is no indexin or ordering. You have to enumerate everything. Thus, it's up to you to figure it out. – Bedplate

Memcache generally aims for O(1) operations. Deleting by wildcard is not O(1), and could potentially lock the server up for relatively long periods of time. See code.google.com/p/memcached/wiki/NewOverview – Eipper

IMO, it sucks that you cant create regions or sections within memcached. App Fabric does that. and instead of storing object, you can store a Type actually. – Fragment

Ugly hack. You're entries are still present. You didn't remove them. Depending on your use case hoping the entries "fall off" in short order won't be sufficient. – Interstate

Why set the namespace key randomly? Why not always start at 0 if it is not present? – Clipclop

@Clipclop - see the answer by ksvendsen below. If it wasn't random, the problem he mentions would be far more likely to occur than 1 in 10,000. – Lamere

I don't understand this solution. Can someone please explain it? does this invalidate all memcache entries stored for foo_namespace_key* ? – Impressible

@Impressible - no, it simply changes the name used for the keys from that point forward. So the old keys are still in memcached, but will end up falling out of the cache eventually since they will no longer be used. – Lamere

@EricPetroelje thank you. I am still confused on how this affect all entries with the namespace. To illustrate I perform ->getAllKeys. This returns an array of keys of the form [0] => sandbox-cache_views ....... [342] => sandbox-cache-schema does the solution above invalidate all these entries if I replace foo_namespace_key with sandbox? if yes does it do it by incrementing exactly what? – Impressible

Answer 2

A note regarding the namespace solution by Eric Petroelje:

Remember that you don't know when memcached will evict you namespace key. Memcache might evict you namespace key, and then when trying to set a new key, it has a probability to 1 to 10000, that it will select the same index key - which means you will get "dirty" results. This is unlikely, but bottom line, it is not secure.

Same problem is with the solution by Poul Vernon.

A safe solution, will to be to use reliable storage (e.g. disk) for the "pointer key"/"namespace key".

Nonparous

Using Memcached::touch will ensure that Paul Vernons's solution will remain in tact. – Johnnajohnnie

@Johnnajohnnie no because we still don't know for sure, when/if memcache will evict the key, the "touch" only "Set a new expiration on an item". And the solution also have problems with race conditions. – Nonparous

Could just use timestamp. Correct about it expiring underneath you, but that's effectively an early/unexpected cache flush. If you can't handle a flush of all related keys, then this solution isn't going to work for you anyway. – Circumgyration

Answer 3

How about this function in php:

function deleteKeysByIndex($search) {
    $m = new Memcached();
    $m->addServer('localhost', 11211);
    $keys = $m->getAllKeys();
    foreach ($keys as $index => $key) {
        if (strpos($key,$search) !== false) {
            $m->delete($key);
        } else {
            unset($keys[$index]);
        }
    }

    // returns an array of keys which were deleted
    return $keys;
}

Deletes keys beginning with $prefix and returns a list of all keys removed. I ran this on 30,000+ keys just now on a shared server and it was pretty quick - probably less than one second.

Serilda

Could also be achieved by storing the matching keys into an array and using $m->deleteMulti($keys) at the end.php.net/manual/en/memcached.deletemulti.php – Ulyanovsk

The problem with that is you'd need to be doing it all in one script or session. The idea here is that I can run this code on a server at anytime from anywhere and it removes the keys. – Serilda

In that case, deleting them separately makes perfect sense. With your approach, one can cancel the execution and restart it whenever desired. I'm actually wondering if the delete and deleteMulti makes any difference in terms of performance, but I think you could only really observe any difference with 1M+ keys. – Ulyanovsk

Use with caution, 1 second for cache invalidation can be too long even for small-medium systems and this approach is not scaling at all. – Colleencollege

memcached getAllKeys doesn't guarantee to return all keys. php.net/manual/en/memcached.getallkeys.php – Natalienatalina

is this not equivelent to to if (strpos($key,$search) === 0) { $m->delete($key);} – Impressible

@Impressible - no. that would require the key begins with $search. this deletes keys that contain $search in any position. – Serilda

@billynoah yup I see that. Thx. I am facing a similar issue and I needed to only drop the ones that begin with a key prefix. And I don't understand why ur solution is not voted up . It is, imo, much clearer than the top answer. – Impressible

Well as others have pointed out the getAllKeys() function is a bit iffy. Also, for the record, it's broken in latest version of libmemcached (bugs.launchpad.net/libmemcached/+bug/1534062). If you are depending on that function to work, make sure you are using memcached 1.4.22 or earlier. – Serilda

@billynoah thanks. After I tried it and it worked for me, I realized that my hosted server has memcache extension. posted a question here #40495869 – Impressible

Answer 4

Create a memcache entry for "1234" and within it store an array of the associated keys. On your delete routine read and iterate through those keys to delete.

Disproportion

I've tried this method before, but it doesn't work under heavy loads since the deletion of all the keys in the associated array is not fast enough. – Leodora

+1 for the only valid solution. The other requires a hope and a prayer (no rand collisions and timely entry dropping). – Interstate

Regarding ksvendsen's reply, using Memcached::touch will ensure that the solution will remain in tact. – Johnnajohnnie