How does the JVM decided to JIT-compile a method (categorize a method as "hot")?

Question

I already worked with -XX:+PrintCompilation, and I know the basic techniques of the JIT-compiler and why JIT-compilation is used.

Yet I still have not found out how the JVM decides to JIT-compile a method, i.e. "when the right time has come to JIT-compile a method".

Am I right in the assumption that every method starts being interpreted, and as long as it is not categorized as "hot method" it will not be compiled? I have something in the back of my head that I read that a method is considered "hot" when it was executed at least 10.000 times (after interpreting the method 10.000 times, it will be compiled), but I have to admit that I am not sure about this or where I've read this.

So to sum up my question:

(1) Is every method interpreted as long as it not has been categorized as "hot" method (and therefore has been compiled) or are there reasons for methods to get compiled even if they are not "hot"?

(2) How does the JVM categorize methods into "non-hot" and "hot" methods? Number of execution? Anything else?

(3) If there are certain thresholds (like number of executions) for "hot" methods, are there Java flags (-XX:...) to set this thresholds?

Underwing

Have a look at the output of -XX:+PrintFlagsFinal, there are lots of flags relating to the JIT compilers and their tiers, inlining, methods sizes, compiler threads etc. – Crash

Answer 1

HotSpot compilation policy is rather complex, especially for Tiered Compilation, which is on by default in Java 8. It's neither a number of executions, nor a matter of CompileThreshold parameter.

The best explanation (apparently, the only reasonable explanation) can be found in HotSpot sources, see advancedThresholdPolicy.hpp.

I'll summarize the main points of this advanced compilation policy:

• Execution starts at tier 0 (interpreter).
• The main triggers for compilation are
  1. method invocation counter i;
  2. backedge counter b. Backward branches typically denote a loop in the code.

• Every time counters reach certain frequency value (TierXInvokeNotifyFreqLog, TierXBackedgeNotifyFreqLog), a compilation policy is called to decide what to do next with currently running method. Depending on the values of i, b and current load of C1 and C2 compiler threads it can be decided to

  • continue execution in interpreter;
  • start profiling in interpreter;
  • compile method with C1 at tier 3 with full profile data required for futher recompilation;
  • compile method with C1 at tier 2 with no profile but with possibility to recompile (unlikely);
  • finally compile method with C1 at tier 1 with no profile or counters (also unlikely).

  Key parameters here are TierXInvocationThreshold and TierXBackEdgeThreshold. Thresholds can be dynamically adjusted for a given method depending on the length of compilation queue.

• Compilation queue is not FIFO, but rather a priority queue.

• C1-compiled code with profile data (tier 3) behave similarly, except that thresholds for switching to the next level (C2, tier 4) are much bigger. E.g. an interpreted method can be compiled at tier 3 after about 200 invocations, while C1-compiled method is subject for recompilation at tier 4 after 5000+ invocations.

• A special policy is used for method inlining. Tiny methods can be inlined into the caller even if they are not "hot". A bit larger methods can be inlined only if they are invoked frequently (InlineFrequencyRatio, InlineFrequencyCount).

Eurus

This link is really helpful and contains everything I wanted to know. You are right, it is hard to find reasonable information out there, haven't read anything that detailed about tiered compilation yet, on most probably never will. Thank you! – Underwing

The key insight for me is that the normal path is 0 (interpreted) -> 3 (C1, full profiling) -> 4 (C2). On that path, C1 really only exists to collect profile data for C2 to work with. – Indefinite

There are then three minor alternative paths. (1) If the C1 compile finds the method is trivial, it's compiled at 1 (C1, no profiling) because 4 (C2) wouldn't be any faster. (2) If the C2 compiler is busy, the method is compiled at 2 (C1, light profiling) until C2 is less busy, at which point it is recompiled at 3 C1, full profiling), so it can carry on to 4 (C2). (3) If C1 is busy but C2 is not, profiling is done in the interpreter, so the method can go straight to C2 without going via C1. – Indefinite

@Eurus it turns out that the general knowledge (mine included) of 10_000 invocations is way wrong. Thank you for the great answer. – Hest

In case anybody else is looking for the current implementation: the AdvancedThresholdPolicy has since been merged into the SimpleThresholdPolicy. – Ret

> finally compile method with C1 at tier 1 with no profile or counters < Does this mean that tier 1 methods do not have the possibility to recompile? Will they recompile when they hit tier 4 thresholds to ensure they get C2 optimizations? Thanks for spelling all of this out - it's nearly impossible to find information on! – Saimon

@Saimon Compiling method at tier 1 implies that HotSpot assumes the method will not benefit from compilation at a higher tier. Usually these are small simple methods that do not need complicated C2 optimizations. – Eurus

Answer 2

The main parameter to control this is -XX:CompileThreshold=10000

Hotspot for Java 8 now uses a tiered compilation by default using a number of stages of compilation from level 1 to 4. I believe 1 is no optimisation. Level 3 is C1 (based on the client client) and Level 4 is C2 (based on the server compiler)

This means that a little optimisation can happen earlier than you might expect and it can keep optimising long after it has reach the 10K threshold. The highest I have seen is escape analysis eliminating a StringBuilder after one million calls.

Note: a loop iterating many times can trigger the compiler. e.g. a loop of 10K times can be enough.

1) Until a method is considered hot enough, it is interpreted. However some JVMs (e.g. Azul Zing) can compile methods on start up and you can force the Hotspot JVM to compile a method via an internal API. Java 9 may also have an AOT (Ahead Of Time) compiler but it is still being researched AFAIK

2) Number of calls, or number of iterations.

3) Yes -XX:CompileThreshold= being the main one.

Impoverish

Do you have further material where I could read up on this? I know about C1 and C2, yet I haven't heard of the compilation levels 1-4 (Interpeter, C1, C2 would be 3 levels). And, you basically say, that a method may be considered hot below the CompilerThreshold. What is this threshold for then? May I excpect that a method is considered hot at the latest when it was called CompilerThreshold times? – Underwing

Sorry for the follow-up question, but just to get my terminology right: May I refer to a method as "hot" if it has been optimized at least one time, or that exactly is the definition of "hot method"? – Underwing

@MarkusWeninger This is something which changes from update to update. If you call the same set of code many time (but each once) you can see they don't all get compiled at once. The actual algo is much more complex. The best place for more information I suspect is the source. Anything else written could be out of date for the current JVM. – Impoverish

@MarkusWeninger compilation is done in the background and if there is lots of method to compile this can be a non-trivial amount of time. I would say a method is "hot" when it becomes a candidate for compilation. Exactly when it gets compiled and replaces existing code is a bit random and hard to control. – Impoverish

Okay, so then thanks for your information! I will leave this question open for some more time to see if further answers are posted, otherwise I will accept yours. :) – Underwing