Maximum size of object that can be saved in memcached

Question

I need to return a rather big file (11MB) to the user. For certain reasons, I can't just provide a direct url to the file (http://www.sample.com/mybigfile.exe); instead it must be accessed through code.

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know). Everything seems to work, fine (no errors), but when I try to retrieve the file from memcached I always get None, as if the file wasn't cached.

Is there a size limit to what can be saved?

Here's the code:

def download_demo():
    """
    Returns the demo file
    """
    KEY = "xyz"
    TIME = 86400 #24 hours

    buff = memc.get(KEY)
    if not buff: 
        file = open(FILENAME, 'r')
        buff = file.read()
        memc.set(KEY, buff, TIME)

    print "Content-Type:application/x-download\nContent-Disposition:attachment;filename=%s\nContent-Length:%s\n\n%s" %    (os.path.split(FILENAME)[-1], len(buff), buff)

Mailbox

If memcache is running locally you will probably not really win much by reading the content from it, since, assuming there is enough RAM available, the file system will already have it cached in RAM. – Inherited

Answer 1

There are two entries about that in the memcached FAQ :

• What is the maximum size of an object you can store in memcached? Is that configurable?

The answer to the first one is (quoting, emphasis mine) :

The maximum size of a value you can store in memcached is 1 megabyte. If your data is larger, consider clientside compression or splitting the value up into multiple keys.

So I'm guessing your 11MB file is quite too big to fit in one memcached entry.

Increasing the size of an object is possible, as per other answers.

Wamble

Note: Memcached now allows you to change the size limit at launch – Tunnell

These links are all dead now – Pangenesis

In newer versions of Memcached you can set the limit up to 128 mb using the -I option (linux.die.net/man/1/memcached). You might consider switching to Redis for large object storing. – Dodge

Wow! The answer is so old that these links died twice. – Brazier

Answer 2

As of memcache 1.4.2, this is a user-configurable parameter: ReleaseNotes142 * memcached @ Github

Configurable maximum item size

Many people have asked for memcached to be able to store items larger than 1MB, while it's generally recommended that one not do this, it is now supported on the commandline.

A few enlightened folk have also asked for memcached to reduce the maximum item size. That is also an option.

The new -I parameter allows you to specify the maximum item size at runtime. It supports a unit postfix to allow for natural expression of item size.

Examples:

memcached -I 128k # Refuse items larger than 128k. 
memcached -I 10m  # Allow objects up to 10MB

Thoughtout

Like in the other answer, these links are also dead now – Embosser

Answer 3

To summarise the necessary steps:

1) Update Memcache to version 1.4.2 or later.

2) Add the flag -I 15M (or however many megabytes) to your memcache run command.

That's either command line, or in Ubuntu, add the line

-I 15M

to anywhere in /etc/memcached.conf and restart the service.

3) Add the necessary flag to the client in memcache.

import memcache
memc = memcache.Client(['localhost'], server_max_value_length=1024*1024*15)
memc.set(KEY, buff, TIME)

If you don't have direct access to the memcache client (i.e. working through a framework), then just hack the memcache code directly.

On Ubuntu, it's /usr/local/lib/python2.7/dist-packages/memcache.py. Change the line:

SERVER_MAX_ITEM_LENGTH = 1024 * 1024 

to

SERVER_MAX_ITEM_LENGTH = 1024 * 1024 * 15

Obviously, you'll need to do this hack over again if you update memcache, but it's a very simple and quick fix.

Uranalysis

The argument you want to pass to the python memcache Client is actually server_max_value_length see memcache.py:163 – Unfurl

Answer 4

The maximum data size is 1mb per item

http://code.google.com/p/memcached/wiki/FAQ#What_is_the_maximum_data_size_you_can_store?_%281_megabyte%29

Update: This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 memcached can store 128mb instead of 1mb (but the developers didn't bother to update the official FAQ). The only reference someone could find id an obscure page that probably will be dead in one year.

Crymotherapy

This is a dead link now – Dodge

I can't find any new source there. Looks like it was bad advertising? – Crymotherapy

In newer versions of memcached you can set the limit to 128 mb using the -I option (linux.die.net/man/1/memcached). I personally switch to Redis for large object storage. – Dodge

This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 some morons are downvoting all answers because (five years after the fact) memcached can store 128mb instead of 1mb and the developers didn't bother to update the official FAQ. The only reference someone could find id an obscure page that probably will be dead in one year. Way to go. And now this is MY fault? – Crymotherapy

@TheDisintegrator just keep your answer updated and be polite. Focus on the content, not the person. Code of Conduct. – Brazier

Answer 5

Following article explained why it limit max size of a single file to 1M.

http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/

Basicly, there are small pieces of memory pages in memcache, inside which objects reside.

And the default page size seems to be 1M, so the max object a page can contain is limited to 1M.

Even though you can config it to increase the size, but I think this might have some kind of performance trade-off.

Peppers

Answer 6

If you are using python memcached client, please also make sure to increase the limit in client memcache.py file along with the memcached server.

~/anaconda/lib/python2.7/site-packages/memcache.py

SERVER_MAX_KEY_LENGTH = 250 
SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 15

you can see in memcached console ( telnet localhost 11211 )

stats slabs STAT 48:chunk_size 3677344

Arvonio

Answer 7

Nobody seems to address this part of the question:

Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know).

This is not a good idea. All modern file systems have efficient caching mechanisms. Reading a file from disk that was read a few seconds earlier will typically still be in memory cache. This is much faster than accessing memcached over a network.

If you are running memcached on 127.0.0.1, the memory it claimed will only be used for that single file, wereas if you simply rely on the caching of your operating system the memory can be used for various purposes depending on the jobs at hand.

Yeomanly

+1. You're right to note Memcache can be unnecessary here. But I will say "it depends". I already work with distributed cache to speed up access to a (file) system. It was very complex, but it may be a good trade-off depending on the use case (number of requests, size of the data, SLA of the system, cost, ...) – Raynata

Answer 8

2022 Update - For open source Memcached,

• the minimum value for max-item-size(-I, --max-item-size) is 1KiB
• the maximum value is 1 GiB
• the default value is 1 MiB

https://github.com/memcached/memcached/blob/597645db6a2b138710f01ffe5e92e453117b987a/memcached.c#L5213

if (unit == 'k' || unit == 'K')
    size_max *= 1024;
if (unit == 'm' || unit == 'M')
    size_max *= 1024 * 1024;

https://github.com/memcached/memcached/blob/597645db6a2b138710f01ffe5e92e453117b987a/memcached.c#L5730

if (settings.item_size_max > (ITEM_SIZE_MAX_UPPER_LIMIT)) {
    fprintf(stderr, "Cannot set item size limit higher than a gigabyte.\n");
    exit(EX_USAGE);
}

In case of cloud memcache variant this value will differ - For example here is the limit on google cloud memcache - gcloud docs

Note: If you are planning to increase the max-item-size value then inspect/increase the amount of memory allocated for the cache(maxbytes : this is the overall Memcached memory). use the -m option to specify the amount of RAM to be allocated (in megabytes). Default is 64MB.

Both settings are available to inspect in memcached statistics (size in bytes):

stats settings
STAT maxbytes 67108864
STAT item_size_max 1048576

Nonsuch