How do I efficiently cache objects in Java using available RAM?

Question

I need to cache objects in Java using a proportion of whatever RAM is available. I'm aware that others have asked this question, but none of the responses meet my requirements.

My requirements are:

• Simple and lightweight
• Not dramatically slower than a plain HashMap
• Use LRU, or some deletion policy that approximates LRU

I tried LinkedHashMap, however it requires you to specify a maximum number of elements, and I don't know how many elements it will take to fill up available RAM (their sizes will vary significantly).

My current approach is to use Google Collection's MapMaker as follows:

Map<String, Object> cache = new MapMaker().softKeys().makeMap();

This seemed attractive as it should automatically delete elements when it needs more RAM, however there is a serious problem: its behavior is to fill up all available RAM, at which point the GC begins to thrash and the whole app's performance deteriorates dramatically.

I've heard of stuff like EHCache, but it seems quite heavy-weight for what I need, and I'm not sure if it is fast enough for my application (remembering that the solution can't be dramatically slower than a HashMap).

Aixlachapelle

what kind of objects are you caching? I'm not quite why you are concerned about the performance of the cache, considering as soon as you are after an expiry policy, you are going to incur more overhead than a plain Map and EHCache is a well developed caching lib, which (I'm thinking of configured via Spring here) is not complex to set up and just as simple to use than a map. – Grice

The objects vary in size from about 1kb to perhaps 10kbs. I'm concerned about the performance because retrieving objects from the cache is in the inner loop of a very CPU intensive process. If it is slow, it can increase the time required for my app to do its thing from minutes to hours. – Aixlachapelle

With softKeys() you won't get a hit if you use equals(), you'll only get a hit if you're looking up the object with reference equality. If you need equals() for cache hits, then use softValues() instead. – Strode

Possible duplicate of Easy, simple to use LRU cache in java – Johnajohnath

Answer 1

I've got similar requirements to you - concurrency (on 2 hexacore CPUs) and LRU or similar - and also tried Guava MapMaker. I found softValues() much slower than weakValues(), but both made my app excruciatingly slow when memory filled up.

I tried WeakHashMap and it was less problematic, oddly even faster than using LinkedHashMap as an LRU cache via its removeEldestEntry() method.

But by the fastest for me is ConcurrentLinkedHashMap which has made my app 3-4 (!!) times faster than any other cache I tried. Joy, after days of frustration! It's apparently been incorporated into Guava's MapMaker, but the LRU feature isn't in Guava r07 at any rate. Hope it works for you.

Arellano

Answer 2

I've implemented serval caches and it's probably as difficult as implementing a new datasource or threadpool, my recommendation is use jboss-cache or a another well known caching lib. So you will sleep well without issues

Spite

Answer 3

I've heard of stuff like EHCache, but it seems quite heavy-weight for what I need, and I'm not sure if it is fast enough for my application (remembering that the solution can't be dramatically slower than a HashMap).

I really don't know if one can say that EHCache is heavy-weight. At least, I do not consider EHCache as such, especially when using a Memory Store (which is backed by an extended LinkedHashMap and is of course the the fastest caching option). You should give it a try.

Snooperscope

This solution shares the same problem that I mentioned with LinkedHashMap, namely (from the EHCache docs) "All caches specify their maximum in-memory size, in terms of the number of elements, at configuration time." – Aixlachapelle

@Aixlachapelle I see... but if you don't want to fill all the available memory with your cached objects (and thus avoid GC), then I don't see how to do that without limiting the number of objects in the cache. Did you do some representative benchmarks to get an idea of the size in memory for different limits? – Snooperscope

I will probably end up benchmarking it if no better option presents itself, but it worries me because reality may differ from my benchmarks in unexpected ways causing all kinds of headaches. – Aixlachapelle

@Aixlachapelle EHCache now seems to support specifyng cache size in bytes. From a very recent EHCache release info (15 Nov 2011): "Tuning cache sizes is now as simple as setting the maximum number of bytes. No more setting maximum entry counts and juggling eviction parameters to approximate the maximum amount of system memory your cache can use." – Follow

Answer 4

I believe MapMaker is going to be the only reasonable way to get what you're asking for. If "the GC begins to thrash and the whole app's performance deteriorates dramatically," you should spend some time properly setting the various tuning parameters. This document may seem a little intimidating at first, but it's actually written very clearly and is a goldmine of helpful information about GC:

https://www.oracle.com/technetwork/java/javase/memorymanagement-whitepaper-150215.pdf

Moulder

I am familiar with that document, but I'm aware of nothing in it that solves my problem :-( – Aixlachapelle

Answer 5

I don't know if this would be a simple solution, especially compared with EHCache or similar, but have you looked at the Javolution library? It is not designed for as such, but in the javolution.context package they have an Allocator pattern which can reuse objects without the need for garbage collection. This way they keep object creation and garbage collection to a minimum, an important feature for real-time programming. Perhaps you should take a look and try to adapt it to your problem.

Plasmasol

Answer 6

This seemed attractive as it should automatically delete elements when it needs more RAM, however there is a serious problem: its behavior is to fill up all available RAM

Using soft keys just allows the garbage collector to remove objects from the cache when no other objects reference them (i.e., when the only thing referring to the cache key is the cache itself). It does not guarantee any other kind of expulsion.

Most solutions you find will be features added on top of the java Map classes, including EhCache.

Have you looked at the commons-collections LRUMap?

Note that there is an open issue against MapMaker to provide LRU/MRU functionality. Perhaps you can voice your opinion there as well

Boarfish

LRUMap requires you to specify the maximum size of the Map, but I don't know what this will be - I just want the cache to keep growing to fill otherwise unused RAM. – Aixlachapelle

Answer 7

Using your existing cache, store WeakReference rather than normal object refererences.

If GC starts running out of free space, the values held by WeakReferences will be released.

Leund

would that approximate LRU though? – Grice

You are thinking of SoftReference - weakly referenced objects can be reclaimed at any time, with no regard to free memory. – Hi

See this talk on the differences between weak and soft references: crazybobs-talks.googlecode.com/svn/trunk/out/references.pdf and parleys.com/#st=5&id=2657&sl=53 . – Strode

Answer 8

In the past I have used JCS. You can set up the configuration to try and meet you needs. I'm not sure if this will meet all of your requirements/needs but I found it to be pretty powerful when I used it.

Lacee

Answer 9

You cannot "delete elements" you can only stop to hard reference them and wait for the GC to clean them, so go on with Google Collections...

Morelli

Answer 10

I'm not aware of an easy way to find out an object's size in Java. Therefore, I don't think you'll find a way to limit a data structure by the amount of RAM it's taking.

Based on this assumption, you're stuck with limiting it by the number of cached objects. I'd suggest running simulations of a few real-life usage scenarios and gathering statistics on the types of objects that go into the cache. Then you can calculate the statistically average size, and the number of objects you can afford to cache. Even though it's only an approximation of the amount of RAM you want to dedicate to the cache, it might be good enough.

As to the cache implementation, in my project (a performance-critical application) we're using EhCache, and personally I don't find it to be heavyweight at all.

In any case, run several tests with several different configurations (regarding size, eviction policy etc.) and find out what works best for you.

Hypogastrium

Answer 11

Caching something, SoftReference maybe the best way until now I can imagine.

Or you can reinvent an Object-pool. That every object you doesn't use, you don't need to destroy it. But it to save CPU rather than save memory

Moonshiner

Answer 12

Assuming you want the cache to be thread-safe, then you should examine the cache example in Brian Goetz's book "Java Concurrency in Practice". I can't recommend this highly enough.

Trophozoite

That didn't really address his question at all. – Hi

@danben: I agree. But thread-safety is something that he'll have to consider. – Trophozoite