PermGen problems with Lift and Jetty

Question

I'm developing on the standard Lift platform (maven and jetty). I'm repeatedly (once every couple of days) getting this:

Exception in thread "7048009@qtp-3179125-12" java.lang.OutOfMemoryError: PermGen space
2009-09-15 19:41:38.629::WARN:  handle failed
java.lang.OutOfMemoryError: PermGen space

This is in my dev environment. It's not a problem because I can keep restarting the server. In deployment I'm not having these problems so it's not a real issue. I'm just curious.

I don't know too much about the JVM. I think I'm correct in thinking that permanent generation memory is for things like classes and interned strings? What I remember is a bit mixed up with the .NET memory model...

Any reason why this is happening? Are the defaults just crazily low? Is it to do with all the auxiliary objects that Scala has to create for Function objects and similar FP things? Every time I restart Jetty with newly written code (every few minutes) I imagine it re-loads classes etc. But even so, it cant' be that many can it? And shouldn't the JVM be able to deal with a large number of classes?

Cheers

Joe

Closeup

As mentioned in the web.archive.org/web/20150105090518/http://www.scala-blogs.org/… article: "The JVM's generational garbage collector is optimized for situations like this and because the implicitly created objects are short-lived, they'll GC very nicely.". but if those objects are not short-lived, that is another story. – Delayedaction

Answer 1

From this post:

This exception occurred for one simple reason :
the permgenspace is where class properties, such as methods, fields, annotations, and also static variables, etc. are stored in the Java VM, but this space has the particularity to not being cleaned by the garbage collector. So if your webapp uses or creates a lot of classes (I’m thinking dynamic generations of classes), chances are you met this problem. Here are some solutions that helped me get rid of this exception :

• -XX:+CMSClassUnloadingEnabled : this setting enables garbage collection in the permgenspace
• -XX:+CMSPermGenSweepingEnabled : allows the garbage collector to remove even classes from the memory
• -XX:PermSize=64M -XX:MaxPermSize=128M : raises the amount of memory allocated to the permgenspace

May be this could help.

Edit July 2012 (almost 3 years later):

Ondra Žižka comments (and I have updated the answer above):

JVM 1.6.0_27 says: Please use:

• CMSClassUnloadingEnabled (Whether class unloading enabled when using CMS GC)
• in place of CMSPermGenSweepingEnabled in the future

See the full Hotspot JVM Options - The complete reference for mroe.

Delayedaction

Thanks. I figured that either adjusting the behaviour of the GC or size of PermGen would solve the problem. I'm really looking for a specific reason why (/if) Lift would balloon PermGen objects. – Closeup

May be because of some dynamic class generation performed by Lift. – Delayedaction

JVM 1.6.0_27 says: Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future. – Autobus

@OndraŽižka good point. I have updated the answer and have added a link to all JVM options for future reference. – Delayedaction

@gyozokudor you mean you cannot revert to the previous version of this answer in order to cancel your edit? – Delayedaction

I rolled back my edit. I couldn't roll back after I edited it because it had to bee reviewed. – Watanabe

No, I rolled back my own edit. For me it shows the original version before I made the edit. I guess maybe the rollback needs to be approved too, I don't know. If it isn't visible probably you could edit it back. I'm sorry for the inconvenience. – Watanabe

Answer 2

If you see this when running mvn jetty:run, set the MAVEN_OPTS.

For Linux:

export MAVEN_OPTS="-XX:+CMSClassUnloadingEnabled -XX:PermSize=256M -XX:MaxPermSize=512M"
mvn jetty:run

For Windows:

set "MAVEN_OPTS=-XX:+CMSClassUnloadingEnabled -XX:PermSize=256M -XX:MaxPermSize=512M"
mvn jetty:run

Should be fine now. If not, increase -XX:MaxPermSize.

You can also put these permanently to your environment.

• For Linux, append the export line to ~/.bashrc

• For Windows, press Win-key + PrintScreen, and go Advanced > Environment. See also http://support.microsoft.com/kb/310519.

Ligature

Answer 3

This is because of the reloading of classes as you suggested. If you are using lots of libraries etc. the sum of classes will rapidly grow for each restart. Try monitoring your jetty instance with VisualVM to get an overview of memory consumption when reloading.

Ikeikebana

Answer 4

The mailing list (http://groups.google.com/group/liftweb/) is the official support forum for Lift, and where you'll be able to get a better answer. I don't know the particulars of your dev setup (you don't go into much detail), but I assume you're reloading your war in Jetty without actually restarting it. Lift doesn't perform dynamic class generation (as suggested by VonC above), but Scala compiles each closure as a separate class. If you're adding and removing closures to your code over the course of several days, it's possible that too many classes are being loaded and never unloaded and taking up perm space. I'd suggest you enable the options JVM options mentioned by VonC above and see if they help.

Frippery

Thank you. I do usually post to the Google group but I thought I'd give StackOverflow a go. Will try increasing the numbers. – Closeup

Answer 5

The permanent generation is where the JVM puts stuff that will probably not be (garbage) collected like custom classloaders.

Depending on what you are deploying, the perm gen setting can be low. Some applications and/or containers combination do contain some memory leaks, so when an app gets undeployed sometimes some stuff like class loaders are not collected, resulting in filling the Perm Space thus generating the error you are having.

Unfortunately, currently the best option in this case is to max up the perm space with the following jvm flag (example for 192m perm size):

-XX:MaxPermSize=192M (or 256M)

The other option is to make sure that either the container or the framework do not leak memory.

Laryngoscope

There needs to be an '=' between MaxPermSize and 192m. – Kenzie