Storing large objects in Memcached

Question

I'm storing an object of size 7-10MB in Memcached and just after putting the object, trying to retrieve it. I get cache misses in that case. Any idea why? This solution works for smaller object sizes.

Background Info:

I'm using Memcached to store a set of large objects of approximately 7-10 MB size. For some reason, it's not possible for me to split this object into multiple smaller keys. I want the cache to be redundant and warm, and hence, I use a slightly complicated cache put procedure, as described below:

keySet = makeRedundantKeys(key) // Appends a unique num to the key
putAsync(keys in keyset)
while(!timeout || countNonNullKeys > desiredQuorumOfKeys) {
    countNonNullKeys = getSyncKeys(key in keySet)
    sleep(backoffTime);
}

I'm getting a lot of failures where getSyncKeys is taking about 700ms to get just one key. Any idea why this might happen? This is observed only for the large objects. Smaller objects <1MB work fine and return the data in ~2ms pAvg. These are good m4.2xlarge EC2 hosts with high network performance. Also, my TCP Retransmitted segment graph spikes up to 1500/min, which seems fishy.

Downrange

Answer 1

By default, memcached will only store objects up to 1MB by default:

Many people have asked for memcached to be able to store items larger than 1MB, while it's generally recommended that one not do this, it is now supported on the commandline.

you can use the -I option to increase this.

In your case you'd need to set -I to be 10m. You're using AWS, so if you're running your own memcached server, you'll be able to do this yourself, but if you're using AWS Elasticache, you'll need to create a parameter group, change the max_item_size from 1MB (1048576) to 10MB (10485760) and apply it to the Elasticache cluster, which needs a reboot.

Gusher

im using php, do i also need to edit chunk_size in php.ini? – Antho

Not, but if you've just got the default value (8k) then it's worth increasing the chunk_size anyway, as you'll need more network writes to transfer anything bigger than that: php.net/manual/en/memcache.ini.php#ini.memcache.chunk-size Data will be transferred in chunks of this size, setting the value lower requires more network writes. Try increasing this value to 32768 if noticing otherwise inexplicable slowdowns – Gusher