How can I determine why the Hotspot JVM decided to re-compile already JIT:ed code a second time?

#Warmup happens here 12893 2351 my.test.application.hotSpot (355 bytes) #Real user logs on here 149755 2351 made not entrant my.test.application.hotSpot (355 bytes) 151913 2837 my.test.application.hotSpot (355 bytes) 152079 2351 made zombie my.test.application.hotSpot (355 bytes)

For posterity, it was fairly simple once I read some of the source code of the hotspot JVM.

The following flags would point out the exact source code line that lead to a function being de-optimized and re-compiled:

-XX:+TraceDeoptimization -XX:+WizardMode -XX:+PrintNativeNMethods -XX:+PrintDependencies -XX:+DebugDeoptimization -XX:+LogEvents

Usually it was an if-statement like this.

void function (Object object){
    if ( object == null ){
        // do some uncommon cleanup or initialization
    }
    do_stuff();
}

Let's say my warmup code never triggered the if statement.

I had assumed that the whole function would be compiled in one go, however, when the JIT C2 compiler actually does decide to produce native code for this function, it will not generate any code for the if-statement because that code path has never been taken.

It will only generate a conditional branch that generates a trap and exception handler in the C2 compiler thread. I think this happens because the native code cache was/is fairly small and so the JVM writer did not want to fill it with potentially useless code.

Anyway, if the statement is ever true (i.e the object is ever null), then the function will immediately and unconditionally trigger this exception handling and be re-compiled ( leading to a freeze/latency hit in the order of a couple ms ).

Of course my warmup code would not call each function in the exact same way as production and I would venture to guess that in any complex product this is close to impossible and a maintenance nightmare anyway.

What this means is that for effectively warming up a java application, every single if-statement in the code needs to be called by th warmup code.

And so we are going to simply abandon the idea of "warming up" our java code because it is not as simple as some would believe.

For the following reasons, we are going to re-write parts of the application to support being ran for weeks/months at a time:

Easier maintenance (we don't need to simulate production during warmup and keep it updated)
The JIT will not be done according to our plastic simulations but instead production behaviour (i.e. use the JIT for what it was designed for instead of fighting it)

Long-term the customer will likely pay for a rewrite in C/C++ or the like to get consistently low-latency but that's for another day.

EDIT: Let me just add that updating to a newer version of the hotspot JVM or "tuning" around the hotspot JVM parameters will never resolve this issue. They are both smokes and mirrors. The fact is that the hotspot JVM was never written for predictable low-latency and this shortcoming is impossible to work around from within the java userland.

Recommended topics

Hot tags