Classloader is not garbage collected, even without GC roots

Question

We have a complex application running under Glassfish V2.1.1. In order to be able to load our code dynamically, we have implemented a CustomClassloader which is able to redefine classes. The behaviour is quite easy: when a dynamically loaded class has change, the current instance of the CustomClassloader is "dropped" and a new one is create to redefine the needed classes.

This works well except that after a few number of time the same class has been reloaded (hence each time a new CustomClassloader is created), we get a PermGen space error because the other instances of CustomClassloader are not garbage collected. (There should be only one single instance of this class)

I tried different methods to track down where the leak is:

1. visualvm => I make a heap dump and extract all the instances of the CustomClassloader. I can see that none of them have been finalized. When I check for the nearest GC root, visualvm tells me that there is none (except for the last instance, because it is the 'real' used one).
2. jmap/jhat => It gives me almost the same result: I see all the instances of CustomClassloader and then when I click on the link to see where are the references on one of them, I get a blank page meaning there is none...
3. Eclipse Memory Analyzer Tool => I get a strange result when I run the following OQL query: SELECT c FROM INSTANCEOF my.package.CustomClassloader c There is only one result, indicating there is only one single instance which is obviously not correct.

I also checked this link and implemented some resources release when a new CustomClassloader is created, but nothing changes: the PermGen memory is still increasing.

So I'm probably missing something and the difference between points (1-2) and (3) shows something I do not understand. Where can I look to get an idea of what's wrong ? Since all the tutorials I followed show how to search the leaking references by using the "Search nearest GC root" feature (and in my case there is none), I don't know how I can track the error.

EDIT 1: I uploaded an example of the heap dump here. The ClassLoader not being unloaded can be selected in visualvm with the following query: select s from saierp.core.framework.system.SAITaskClassLoader s One can see that there are 4 instances and the three first should have been collected because there is no GC root... There must be a reference somewhere but I don't know how I can search for it. Any hint is welcomed :)

EDIT 2: After some deeper tests I see a very strange pattern. The leak seems to depend on the data that are being loaded by OpenJPA: if no new data is loaded, then the classloader can be GCed, otherwise it is not. Here is the code I use when a create a new SAITaskClassLoader to 'clear' the old one:

PCRegistry.deRegister(cl);
LogFactory.release(cl);
ResourceBundle.clearCache(cl);
Introspector.flushCaches();

= Pattern 1 (Classloader is GCed): =

1. New SAITaskClassLoader
2. Load Data D1, D2, ..., Dn
3. New SAITaskClassLoader
4. Load Data D1, D2, ..., Dn
5. ...

= Pattern 2 (Classloader is NOT GCed): =

1. New SAITaskClassLoader
2. Load Data D1, D2, D3
3. New SAITaskClassLoader
4. Load Data D3, D4, D5
5. New SAITaskClassLoader
6. Load Data D5, D6, D7
7. ...

In all cases, the SAITaskClassLoader that have been cleared have no GC root. We are using OpenJPA 1.2.1.

Thanks & Best regards

Twain

You tried the Singleton Pattern? – Sacroiliac

Yep, actually the CustomClassloader is a Singleton. – Twain

That's odd. If that was the case then you should never be able to have multiple objects of this type that are never finalized. The first object would stop you...hmm. Have you tried setting the discarded object to null? – Sacroiliac

Yes and this does not change anything. Moreover, in the patterns I put above, I load the same amount of data. Hence the PermGen will increase until there is no new data loaded... – Twain

I have create a mini-project that reproduces de bug here – Twain

Answer 1

Finally, I can close this bug, since it seems to be linked with OpenJPA and non-parametrized queries. Another thread to look at: Custom ClassLoader not garbage collected

Twain

Answer 2

Without snippets of the source code of CustomClassLoader or the actual heap dumps, it will be very hard to track down the problem. Your CustomClassLoader can't be a singleton. If it is, your design can't work (or i missed something).

You need to get the list of ClassLoader instances of type CustomClassLoader and track down the references to these objects.

These posts may help you on how to analyze it further and get into the dark detaisls of hunting down ClassLoader leaks:

• Debugging Dynamic ClassLoaders in Heap Dumps
• How to analyze leaky web-apps
• A couple of typical causes of ClassLoader leaks

Maxwellmaxy

Thanks for your links. When I say the CustomClassloader is singleton, it is true from an external class, the only method to get a reference to it is a static getInstance(). It will be able to see when there is a class change, and create internally a new instance that will be stored in the static internal variable. The latter is the only explicit reference in the whole application. – Twain

Maybe I found something. I'm using OpenJPA 1.2.1 and there is this bug: issues.apache.org/jira/browse/GERONIMO-3326 – Twain

I've uploaded the ZIP of my heap so you can have a look. – Twain

A shot into the blue: there seem to be references from e.g. BusinessTask to SAITaskClassLoader to ResourceBundle. Try using ResourceBundler.clearCache(saiTaskClassLoader) once in a while. Another issue may be java.util.Logger. Glassfish is using its own ServerLogManager, maybe there is sth going wrong? (Heap dump object 0x784de8310 and 0x7070236c8) Perhaps it's this bug: bugs.sun.com/bugdatabase/view_bug.do?bug_id=6717784 – Maxwellmaxy

Thanks for your reply. I already added the ResourceBundle.clearCache, but this changes nothing. Concerning the glassfish bug, in such a case, there would be GC roots displayed in visualvm... I built a stress test that forces the creation of a new SAITaskLoader and I monitored the PermGen: it seems that the leak is linked to the data: if no new data is loaded the classloader can be collected when a new one is created. I saw that the first instances of SAITaskClassLoader are never collected but then most of the others are. But this does not explain why no root GC is show in the heap analyzers :( – Twain

Answer 3

Garbage collection of classloaders is an exceptionally tricky business. Using JProfiler, I see the following chain of incoming references to the currently active custom class loader:

This shows that you have a static field "singleInstance" in your custom classloader that references the class loader itself. You should try to clear that field on redeployment in order to make it easier for the VM to collect the classloader.

A note about the result you got with Eclipse MAT: It removes all objects that are not strongly reachable. JProfiler also does this by default. So the three previous classloaders should be garbage collected but they are not, due to the special rules that the JVM has for classloader GC which are not captured by the standard references in the heap.

Disclaimer: My company develops JProfiler

Helle

Thanks for your reply. However, this is correct :) There is one instance of the SAITaskClassLoader which is held into a static variable. When a class is updated, that instance is dropped and a new one is then stored in this [static] variable. The problem is that the dropped instances are not garbage collected and all the heap analyzers I tested shows that they don't have any GC root... In the dump, 3 of the 4 instances should have been collected, but are never, causing a PermGen OutOfMemoryError after some new instances of the SAITaskClassLoader are created. – Twain

Depending on how you do this, it may still represent a problem. This is not necessarily about the paths to GC roots, classloader GC has different rules in addition to strong reachability and it's all very fragile. Maybe you could insert another layer that is not reloaded itself and hold the singleton instance of the class loader there, then you have more explicit control. – Helle

I've identified a pattern and added some graphs, so maybe someone can have an idea of where to look... – Twain

Do you have any additional information on the topic? I'm facing a similar problem. A classloader and only weak references pointing to him but it doesn't get GCd – Scornful