Memory-aware LRU caching in Python?

Question

I'm using Python 3's builtin functools.lru_cache decorator to memoize some expensive functions. I would like to memoize as many calls as possible without using too much memory, since caching too many values causes thrashing.

Is there a preferred technique or library for accomplishing this in Python?

For example, this question lead me to a Go library for system memory aware LRU caching. Something similar for Python would be ideal.

---

Note: I can't just estimate the memory used per value and set maxsize accordingly, since several processes will be calling the decorated function in parallel; a solution would need to actually dynamically check how much memory is free.

Comate

If you can't find something out there that does this already, you can try leveraging psutil (code.google.com/p/psutil) to roll your own. – Fatimafatimah

Yes, that's what I'm looking into right now—in fact, do you happen to know how to find the source for Python 3's lru_cache implementation? The easiest way would be to simply check the memory usage within the decorator. It would add some overhead for sure, but in this application I don't think it would be significant. – Comate

@Comate for the source see functools.py:370 and a couple lines above for the cache key functions. – Fraktur

Or locally: Launch an interactive Python interpreter, import functools and just enter the module name functools. Works for any locating the source of nearly any Python module (except C extensions of course). – Fraktur

@Comate BTW, functools.lru_cache was written by Raymond Hettinger. He posted several different (LRU) caching / memoization recipes, maybe you can find something useful or at least inspirational in those ;-) – Fraktur

Thanks guys, seems to be working well. Posted the code in my answer. Let me know if it can be improved at all! – Comate

I asked in github.com/tkem/cachetools/issues/152 We will see :) – Kleptomania

Answer 1

I ended up modifying the built-in lru_cache to use psutil.

The modified decorator takes an additional optional argument use_memory_up_to. If set, the cache will be considered full if there are fewer than use_memory_up_to bytes of memory available (according to psutil.virtual_memory().available). For example:

from .lru_cache import lru_cache

GB = 1024**3

@lru_cache(use_memory_up_to=(1 * GB))
def expensive_func(args):
    ...

Note: setting use_memory_up_to will cause maxsize to have no effect.

Here's the code: lru_cache.py

Comate