Java solutions to record / replay executed code in a JVM

Question

We have a bug we're trying to find that happens non-deterministically (well, it's deterministic, but we just don't know what's actually causing it) and it only happens once every couple hours.

We read a lot of network data, and we have many threads, so there's a likelihood it's from an input, race condition, or combination of both, but for the most part, unreproducible.

I'm wondering if there are any JVM recorders / replayers out there that can store everything that happened in a JVM so we can then go through it step by step and recreate the steps to the exception.

I've found one. (I'm not going to post it so people don't think I'm trying to advertise a product.), but I'm wondering if there are others, and more importantly, others that people have used, and can report that work well.

Edit to add:

I've found Replay Solutions. I've never used it, and I've never heard of it, so I don't know how good it actually is.

We use JProfiler, but I don't think it actually has support to record/replay everything.

Luciusluck

go ahead and post the link. It's always ok if it's not your company's solution, and even if it's your company's solution it is ok if you add a disclaimer like you did. – Grigg

Answer 1

I would suggest using Chronon it worked well for me, but I haven't used it extensively yet.

Conciliate

Ahhh....That's the one I saw reviewed months ago. They really need to get their SEO up to date. I couldn't find it for the life of me. – Luciusluck

Do you know if it will support multiple threads? As in will it guarantee that each instruction (even on different threads) will get executed in the same order as it was originally? – Luciusluck

I don't know about that, I would suggest you contact them and discuss your problem and promise that you blog about it. They are still trying to increase their visibility ;) – Substantiate

Quoted from their site @Reverend Gonzo so I guess yes: Does that mean the end of non-reproducible bugs? Yes! You no longer need to ‘reproduce’ bugs, since the recording has captured all the execution and thus the bug with it. You can just play back the recording in the debugger and solve the bug. This is specially helpful with hard to reproduce bugs like race conditions, deadlocks, etc. – Substantiate

As the author of Chronon I can confirm that Chronon does support multiple threads :) – Scevour

Answer 2

It is not possible (except maybe for micro-benchmarks) to record everything, especially because of the thread interleavings: the performance degradation/slowdown would be too severe. If the problem you are trying to reproduce is a concurrency bug you may want to try systems that perform partial logging and complete the interleaving using SMT solving:

http://www.gsd.inesc-id.pt/~nmachado/software/Symbiosis_Tutorial.html

To the extent of my knowledge, this is the most recent system, and is available open-source (in java, c and c++).

Wernick