We use a number of large objects. Ideally, we'd like to make all of them permanently available to client code, but they don't fit in physical memory all at once. So when we approach memory limits, we'll need to destroy some of the objects in pool (probably, we'll destroy the least-recently-used object). (Using virtual memory on disk is tantamount to a complete system freeze.)
For concreteness, the memory per object varies between ~100MB and ~10GB; the total amount of RAM we have is 32GB.
Unfortunately, we don't know the memory used by each object until it's created. Therefore, we cannot predict if any given object would fit into the available physical memory.
A good solution would be to provide a callback function to Python memory allocator, and have memory allocator call it when sees that the amount of free physical memory is below a certain threshold. That callback function would simply destroy an object from the pool, and return. While in general, there's no guarantee that memory would become instantly available, it certainly would in CPython (which releases the memory the moment reference counter hits zero), which is probably as much as I can ask for.
Unfortunately, I'm not aware of any way to have Python memory allocator call a function provided to it when memory runs low. Is there anything else I can do?
I suppose I could have a separate thread running in the background, and checking available memory every second or so. If it sees less than 5GB physical memory available, it would start destroying objects from the pool. Assuming object destruction proceeds is on average at least as fast as object creation, the 5GB cushion won't get used up much, and we'll be fine. This seems pretty fragile; but any implementation ideas for this approach would still be great.
This has to work on Linux under Python 3.2, but would be nice if there was a (possibly different) solution for Windows too.
NOTE: Three tiers of memory allocation are at work: OS, malloc (C runtime), and Python object allocator.