JBoss 7, java.lang.OutOfMemoryError: PermGen space

Question

I've hit this error where the CPU usage goes to its limits and JBoss needs a restart (java.lang.OutOfMemoryError: PermGen space).

I found a solution for older JBoss version to increase the MaxPermSize. I guess the same goes for JBoss7.

Which value would be good enough in order not to face any problem again? Is there any way to get permanently away from this problem (like lets say use a different VM like JRockit)?

Crosslink

This really shouldn't be happening on AS 7 unless you're deploying a lot of big classes. Because of its modular classloader, JBoss AS 7 shouldn't suffer from classloader leaks which - combined with multiple redeploys during testing - are the most common cause of this problem on other servers. Do you get these errors only after lots of redeploys? Under what circumstances? – Wahl

we have been testing our applications and during this phase we had extensive redeploys . And out of nowhere this problem struck – Crosslink

Interestingly enought, I'm running the JBoss 7.1.1 on my windows server as a service. In this scenario, after the deployment it's dead. Running the server from standalone.bat, works solid as on my development machine. – Salop

Answer 1

Since this happens after multiple redeploys it sounds like you've encountered a classloader leak, a common kind of permgen leak.

These lovely beasts happen because of normal (non-weak) references from objects owned by the container to objects that're instances of classes loaded from the application classloader. If those references aren't cleared on undeploy there's still a strong reference chain to the application's classloader, so it can't be GC'd and the loaded classes can't be freed.

A common cause is static collections in container classes that have non-static references to application classes added.

JBoss AS 7 has some fairly strong provisions in its module system to prevent classloader leaks, so I'm surprised you've managed to trigger one. I haven't seen a classloader leak since I moved to AS7 from Glassfish.

Increasing MaxPermSize will buy you some time, but it won't get rid of the problem.

You really need to work out why the classloader is leaking. Doing this is "fun". You enjoy taxes, intermittent faults, and cleaning the shower, right? See the links in the first par for some blogs that'll get you started on tracking the leak down. Basically, you'll want to use VisualVM and OQL to dig for references to your app classloader, or take a heap dump and use jhat (part of the JDK) to find the references. Either way, the idea is to figure out where the strong reference chain from the app server to your classloader via instances of your app classes is.

Alternately, it can help to take a copy of your app then start ripping bits out of it until the leak goes away. You can tell if it's leaking by connecting VisualVM or other monitoring to the app server VM and watching to see if PermGen increases after two or more deploy/undeploy cycles. Consider automating the deploy/undeploy cycles. Narrow down the cause of the leak to a small part of your app and/or one of its dependencies and produce a small, self-contained test case, then submit that as a bug report on (a) JBoss AS 7, since AFAIK it's meant to stop this happening and (b) the culprit that's holding the reference.

If you narrow the cause down to a dependency that's bundled inside your deployment archive, moving it into a JBoss AS 7 module may take care of the problem. Create a JBoss module for it, deploy it to the modules directory of AS7, and add a dependency on it to your deployment via your Manifest.MF or via a jboss-deployment-structure.xml. See the documentation on the AS7 class loader.

This is why the fact that Project Jigsaw has been deferred makes me sad. Java needs a strong module system that gets rid of this crud.

Wahl

jhat refers to the approach proposed by Frank Kieviet. But this approach is not very convenient. A better way is to 1) make a heap dump; 2) open the heap dump in Eclipse MAT; 3) look for the duplicated classes; 4) right-click on the most duplicated classes then select "Merge Shortest Paths to GC Roots" with option "exclude weak references". This gives the list of object which prevent the permgen space to be reduced by class unloading. – Quasijudicial

Answer 2

The VM parameter is:

-XX:MaxPermSize=256M

Just make it large enough that you don't hit the limit.

Broadly speaking, perm gen memory is used for objects associated with Classes and interned Strings. You shouldn't run out unless you have use lot of different classes.

Headlight

do you know the default value of MaxPermSize for JBoss 7? – Crosslink

While useful, this won't help with a classloader leak, only postpone the problem. Eventually you'll run out of Permen space. – Wahl

Out of curiosity what is considered a lot of different classes for a web application, 100, 1000, 10000? – Chondrule

By the way i do this in standalone.conf – Architecture

Answer 3

We had to do the exact same thing and unfortunately visualvm did not do much for us.

We ended up using eclipse mat to analyse the heap dump generated on crash and then checked out the leak suspects report which told us that there were a lot of ModuleClassLoader instances being leaked.

Clicking on one of the instances in the overview tab and then selecting merge shortest paths to GC roots + exclude weak references gave us our culprit which was not allowing these ModuleClassLoader instances to be GC'ed!

https://smalldata.tech/blog/2015/09/29/detecting-java-permgen-memory-leak

Murguia

Answer 4

Why it happens?

The "PermGen" error happens, when the Java virtual machine runs out of memory in the permanent generation. Recall that Java has a generational garbage collector, with four generations: eden, young, old and permanent. In the eden generation, objects are very short lived and garbage collection is swift and often. The young generation consists of objects that survived the eden generation (or was pushed down to young because the eden generation was full at the time of allocation), garbage collection in the young generation is less frequent but still happens at quite regular intervals (provided that your application actually does something and allocates objects every now and then). The old generation, well, you figured it. It contains objects that survived the young generation, or have been pushed down, and garbage collection is even less infrequent but can still happen. And finally, the permanent generation. This is for objects that the virtual machine has decided to endorse with eternal life - which is precicely the core of the problem. Objects in the permanent generation are never garbage collected; that is, under normal circumstances when the jvm is started with normal command line parameters. So what happens when you redeploy your web application is, that your WAR file is unpacked and its class files loaded into the jvm. And here's the thing: almost always ends up in the permanent generation... (taken from: http://rlogiacco.blogspot.com/2009/02/jboss-and-permgen-outofmemoryerror.html)

Here are some advices :

use this parameters for your JVM. they tell the Garbage Collector to invoke its algorithm also on the PermGen.

set JAVA_OPTS=-Xms512m -Xmx1024m 
-XX:PermSize=512m 
-XX:MaxPermSize=1024m 
-XX:+UseConcMarkSweepGC 
-XX:+CMSPermGenSweepingEnabled 
-XX:+CMSClassUnloadingEnabled 

• The CMSPermGenSweepingEnabled setting includes the PermGen in a garbage collection run. By default, the PermGen space is never included in garbage collection (and thus grows without bounds).
• The CMSClassUnloadingEnabled setting tells the PermGen garbage collection sweep to take action on class objects. By default, class objects get an exemption, even when the PermGen space is being visited during a garabage collection.

Restart your JBOSS because each time you deploy application, you increase the amount of data in the PermGen.

You can also use JRocket JVM instead of the Sun JVM. it doesn't have any PermGen in its Garbage Collector algorithm.

Christogram