What makes hot deployment a "hard problem"?

Asked 18/3, 2009 at 23:2 Answered 18/3, 2009 at 23:37

At work, we've been having a problem with "PermGen out of memory" exceptions, with the team lead deciding it was a bug in the JVM - something related to hot-deployment of code. Without explaining many details, he pointed out that hot deployment is a "hard problem", so hard that even .NET doesn't do it yet.

I found a lot of articles explaining hot deployment from the bird's-eye-view, but always lacking technical details. Could anyone point me to a technical explanation, and explain why hot deployment is "a hard problem"?

Shortcoming answered 18/3, 2009 at 23:2 Comment(3)

Why is this Community Wiki, by the way? – Crossruff 19/3, 2009 at 2:16

Thought it might help in case I left something unclear... do that by default, is it bad etiquette? – Shortcoming 19/3, 2009 at 3:56

Funny: found a think-o while re-reading just now, thanks! – Shortcoming 19/3, 2009 at 3:57

When a class is loaded, various static data about the class is stored in PermGen. As long as a live reference to this Class instance exists, the class instance cannot be garbage collected.

I believe that part of the problem has to do with whether or not the GC should remove old Class instances from perm gen, or not. Typically, every time you hot deploy, new class instances are added to the PermGen memory pool, and the old ones, now unused, are typically not removed. By default, the Sun JVMs will not run garbage collection in PermGen, but this can be enabled with optional "java" command arguments.

Therefore, if you hot deploy enough times, you will eventually exhaust your PermGen space.

If your web app does not shut down completely when undeployed -- if it leaves a Thread running, for example -- then all of the Class instances used by that web app will be pinned in the PermGen space. You redeploy and now have another whole copy of all of these Class instances loaded into PermGen. You undeploy and the Thread keeps going, pinning ANOTHER set of class instances in PermGen. You redeploy and load a whole net set of copies... and eventually your PermGen fills up.

You can sometimes fix this by:

Supplying command arguments to a recent Sun JVM to enable GC in PermGen and of classes. That is: -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
Using a different JVM that doesn't employ a fixed sized PermGen or that does GC on loaded classes

But this will help only if your web app shuts down completely and cleanly, leaving no live references to any of the Class instances of any Class loaded by the Class loaders for that Web App.

Even this will not necessarily fix the problem, due to class loader leaks. (As well as too many interned strings in some cases.)

Check out the following links for more (the two bolded ones have nice diagrams to illustrate part of the problem).

Crossruff answered 18/3, 2009 at 23:2 Comment(8)

Eddie -- isn't the reason for keeping the classes though because there are already existing objects knocking around with the "old" class definition? – Wonderstricken 18/3, 2009 at 23:31

@Edie: Most of the links are related to memory rather than classloader architecture which would be most useful to understand why hot-deploy is a problem. The memory problem is a consequence of the workarounds. – Dulcine 18/3, 2009 at 23:44

Just a note about running threads: when I left a thread to sleep over re-deploy by mistake, the moment the thread wake up it thrown a NoClassDefFound (classes were undeployed), and then (of course) died. After it's death PermGen should be OK to be GCed. – Bandore 19/3, 2009 at 1:52

@Vladimir Dyuzhev: Yes, once the Thread dies (and assuming there are no more live references to the Class objects) the Class objects should be GCable in PermGen, if you have it enabled. – Crossruff 19/3, 2009 at 2:38

@Neil Coffey: Of course the reason the classes are kept around is because live objects still reference them. That is the root of the problem! There shouldn't be any live references. If you implement your web application perfectly, then this will not be a problem. But you have to be perfect. – Crossruff 22/2, 2013 at 0:46

@OscarRyz: Yes, the links are related to PermGen memory, which is what the OP said got exhausted. The memory problem is not a consequence of the workarounds, or at least not only. You have to be sure to perfectly stop all threads, release all static references, and so on for your app to undeploy without leaving references to classes in PermGen that would otherwise be GCable. – Crossruff 22/2, 2013 at 0:47

Looking at this in 2015- I wonder how the challenges have changed in Java 8. It's my understanding that permgen space is not an issue in this release – Nazareth 26/2, 2015 at 18:18

@IcedDante: I haven't carefully investigated Java 8, but I imagine this problem is still there in a different form. The fundamental problem of being unable to GC the class objects if you haven't perfectly released all resources on web app shutdown will still be there. Just, the object that is "leaked" won't be in PermGen (which no longest exists) but in the heap. – Crossruff 16/4, 2015 at 23:15

The problem in general terms is the security model of Java that actually attempts to prevent that a class that has already been loaded be loaded again.

Of course Java since the beginning has supported dynamic class loading, what it is difficult is class re-loading.

It was consider harmful ( and for a good reason ) that a running java application got injected with an new class with malicious code. For instance a java.lang.String cracked implementation comming from the internet , that instead of creating string, deletes some random file hile invoking the method length().

So, they way Java was conceived ( and I presume .NET CLR in consequence, because it was highly "inspired" in JVM's ) was to prevent an already loaded class to load again that same VM.

They offered a mechanism to override this "feature". Classloaders, but again the rules for the class loaders was, they should ask permission to the "parent" classloader before attempting to load a new class, if the parent has already loaded the class, the new class is ignored.

For instance I have used classloaders that load a classes from LDAP or RDBMS

The hot deploy becomes a necessity in the Java world when the application server became mainstream for Java EE ( and also create the need for micro containers like spring to avoid these kind of burden ) .

Restarting the whole app server after every compile drives anyone crazy.

So app server provider, offer this "custom" class loaders to help hot deployment, and using a configuration file, that behavior, SHOULD be disabled when set in production. But the tradeoff is you have to use tons of memory in development. So the good way to do this is restart every 3 - 4 deployments.

This doesn't happen with other languages that were designed from the beginning to load their classes.

In Ruby for instance, you can even add methods to a running class, override a method at runtime or even add a single method to an unique specific object.

The tradeoff in these kinds of environments is of course memory and speed.

I hope this helps.

EDIT

I've found this product some time ago that promises that reload is make as simple as possible. I didn't remember the link when I first wrote this answer, and I do.

It is JavaRebel from ZeroTurnaround

Dulcine answered 18/3, 2009 at 23:37 Comment(7)

what a mess in your head, eh? so java wasn't designed to load classes at runtime, uh? that's why it has Class.forName() from the very beginning? – Bandore 18/3, 2009 at 23:49

That isn't what Oscar wrote Vladimir. He wrote that Java wasn't designed to replace classes at runtime. – Wineshop 19/3, 2009 at 0:14

ClassLoader class is there from version 1. Not designed to replace classes at runtime? Besides, security has nothing to do with PermGen OOM. The limit on PermGen is just a nasty implementation detail of Sun JVM. – Bandore 19/3, 2009 at 1:47

Java has a security model where one can choose to prevent some of what you're talking about, but Java has always been able to dynamically load classes. – Crossruff 19/3, 2009 at 2:13

Hey!! I NEVER said Java wouldn't load classes at runtime! What I said, is: "...to prevent an already loaded class to [be] load[ed] again...". Please learn to read ( and I promise fix my writing ). @Valdimir: Try re-loading java.lang.String with your Class.forName(), you totally misread the question – Dulcine 19/3, 2009 at 2:31

The whole point was "hot-deploy" that involves classes reloading. PermGen is not the topic here. Again read the original question: "..why hot deployment is "a hard problem"?.." So any information on PermGen is and additional information. My answers attempts to explain why is that a hard problem. – Dulcine 19/3, 2009 at 2:38

OK, got it. It's just not very clear from the text. :-) You start right from security as if it's the main/only reason for trouble. – Bandore 19/3, 2009 at 12:18

Sun JVM has PermGen space fixed, and eventually it's all consumed (yes, apparently due to a bug in classloader-related code) => OOM.

If you can use another vendor's JVM (e.g. Weblogic one), it dynamically extends PermGen space, so you'll never get permgen-related OOM.

Bandore answered 18/3, 2009 at 23:2 Comment(2)

Wait there. This is not 100% true. The perm gem problem not only related to the JVM and its libraries. Bad application can also cause PermGen errors. Also, dynamically expanding PerGen it is absolutely diffrente from solving it. Wait enough time in any VM and the problem will happen. – Bother 19/3, 2009 at 2:18

Correct. Still, dynamic PermGen allows you to survive times more hot deployments. In Sun JVM it often happens on redeployment #1 :-E – Bandore 19/3, 2009 at 12:16

Which version of java are you using? There were bugs in early Sun 1.4.2, but it's been working for a long long time.
BTW, how will you break the news to your team lead? Are you the team lead?

Vancevancleave answered 18/3, 2009 at 23:2 Comment(2)

It's actually new problem since 1.5. Or at least I ran into it in 1.5 only. – Bandore 19/3, 2009 at 1:46

We're running into this problem with 1.6. I'll probably "break this" to my team lead by sending him a link to this page :-P – Shortcoming 19/3, 2009 at 2:16

Recommended topics

Hot tags