Just in Time compilation always faster?

Question

Greetings to all the compiler designers here on Stack Overflow.

I am currently working on a project, which focuses on developing a new scripting language for use with high-performance computing. The source code is first compiled into a byte code representation. The byte code is then loaded by the runtime, which performs aggressive (and possibly time consuming) optimizations on it (which go much further, than what even most "ahead-of-time" compilers do, after all that's the whole point in the project). Keep in mind the result of this process is still byte code.

The byte code is then run on a virtual machine. Currently, this virtual machine is implemented using a straight-forward jump table and a message pump. The virtual machine runs over the byte code with a pointer, loads the instruction under the pointer, looks up an instruction handler in the jump table and jumps into it. The instruction handler carries out the appropriate actions and finally returns control to the message loop. The virtual machine's instruction pointer is incremented and the whole process starts over again. The performance I am able to achieve with this approach is actually quite amazing. Of course, the code of the actual instruction handlers is again fine-tuned by hand.

Now most "professional" run-time environments (like Java, .NET, etc.) use Just-in-Time compilation to translate the byte code into native code before execution. A VM using a JIT does usually have much better performance than a byte code interpreter. Now the question is, since all an interpreter basically does is load an instruction and look up a jump target in a jump table (remember the instruction handler itself is statically compiled into the interpreter, so it is already native code), will the use of Just-in-Time compilation result in a performance gain or will it actually degrade performance? I cannot really imagine the jump table of the interpreter to degrade performance that much to make up the time that was spent on compiling that code using a JITer. I understand that a JITer can perform additional optimization on the code, but in my case very aggressive optimization is already performed on the byte code level prior to execution. Do you think I could gain more speed by replacing the interpreter by a JIT compiler? If so, why?

I understand that implementing both approaches and benchmarking will provide the most accurate answer to this question, but it might not be worth the time if there is a clear-cut answer.

Thanks.

Cacka

If very aggressive optimization is done before compiling I doubt it. Depends on how good your optimalizations are though. – Ontario

Interesting premise for a Computer Science doctoral thesis. Let us know when the results are published. – Cates

Answer 1

The answer lies in the ratio of single-byte-code-instruction complexity to jump table overheads. If you're modelling high level operations like large matrix multiplications, then a little overhead will be insignificant. If you're incrementing a single integer, then of course that's being dramatically impacted by the jump table. Overall, the balance will depend upon the nature of the more time-critical tasks the language is used for. If it's meant to be a general purpose language, then it's more useful for everything to have minimal overhead as you don't know what will be used in a tight loop. To quickly quantify the potential improvement, simply benchmark some nested loops doing some simple operations (but ones that can't be optimised away) versus an equivalent C or C++ program.

Manlike

Answer 2

JIT can theoretically optimize better, since it has information not available at compile time (especially about typical runtime behavior). So it can for example do better branch prediction, roll out loops as needed, et.c.

I am sure your jumptable approach is OK, but I still think it would perform rather poor compared to straight C code, don't you think?

Esoterica

Or on the other hand, the overhead of collecting the information for then making those optimisations may prove more significant than the then-faster native code.... The theory isn't really one-sided - but you can get some statistical confidence from experience in a particular problem domain. – Manlike

@Tony: You are right - often simpler is better - but this is turning to VM implementation practice, and in many cases works well especially for long running programs. – Esoterica

True. But, if the gains were really significant, then compiled languages would put in native instrumentation code and the already-compiled alternative and mutate themselves when warranted (which of course requires getting the OS to allow writes to executable memory pages). That that's not typically pursued suggests to some extent the VM languages are waving this around more as an theoretical jab at their precompiled counterparts, rather than it being particularly effective...? Or am I too suspicious? ;-) – Manlike

@Tony: I don't think you are following development - LLVM is becoming important and already has frontends for C/C++/Obj-C/Fortran et.c. – Esoterica

Answer 3

When you use an interpreter, the code cache in your processor caches the interpreter code; not the byte code (which may be cached in the data cache). Since code caches are 2 to 3 times faster than data caches, IIRC; you may see a performance boost if you JIT compile. Also, the native, real code you are executing is probably PIC; something which can be avoided for JITted code.

Everything else depends on how optimized the byte code is, IMHO.

Trabeated