Taking thread dumps in production

Question

I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on

1. Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.

2. Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.

Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2

1. Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
2. The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
3. Any more?

I would appreciate to get suggestions/inputs on

1. Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?

2. Any other approach to take thread dumps?

Thank you.

Edit -

A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers. If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.

Dania

Is this in relation to a JEE application? – Uxorial

Answer 1

You can use

jstack {pid} > stack-trace.log

running as the user on the box where the process is running.

If you run this multiple times you can use a diff to see which threads are active more easily.

---

For analysing the stack traces I use the following sampled periodically in a dedicated thread.

 Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();

Using this information you can obtain the thread's id, run state and compare the stack traces.

Heinie

The first point of executing jstack is the same as approach 1 I mention above? Integrating and triggering it through jmx helps the user to not worry about the process id. Also with the jmx approach, we can schedule jstack to be executed repeatedly after every specified interval. Regarding the second point about calling Thread.getAllStackTraces(); this would mean I would have log the thread dump manually in such a way that it gets interpreted by the thread dump analyzers. What is the benefit of this approach over the first one you mention? – Dania

I would have the analysers in the program and only log the information you are interested in. In my case this only occurs when I have a problem which usually means I want to see what other logs are occuring at the time e.g. what time did it occur, what is the last thing that thread logged and was an error produced afterwards. – Heinie

Answer 2

With Java 8 in picture, jcmd is the preferred approach.

jcmd <PID> Thread.print

Following is the snippet from Oracle documentation :

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.

However, shipping this with the application may be licensing implications which I am not sure.

Kellum

That's good to know. Can they be parsed by open source thread dump analyzers like TDA? Do they include the native thread id? – Dania

Answer 3

If its a *nix I'd try kill -3 <PID>, but then you need to know the process id and maybe you don't have access to console?

Noninterference

Yes. Access to the console and getting the process id are issues with the "kill" approach. The approaches I mention above do not have these drawbacks. – Dania

Answer 4

I'd suggest you do all the heap analysis on a staging environment if there is such an env, then reflect your required Application Server tuning on production if any. If you need the dumps for analysis of your application's memory utilization, then perhaps you should consider profiling it for a better analysis.

Heap dumps are usually generated as a result of OutOfMemoryExceptions resulting from memory leaks and bad memory management.

Check your Application Server's documentation, most modern servers have means for producing dumps at runtime aside from the normal cause I mentioned earlier, the resulting dump might be vendor specific though.

Uxorial

I understand the overhead of getting a thread dump but to get a clear understanding of whats happening in production environments we need this feature. We use glassfish application server and getting a thread dump in glassfish is nothing special. You need to send a kill signal to the jvm which has two drawbacks mentioned in my reply to @Peter Lljenbery. – Dania

Got it, then I would recommend you use a special build with memory dump enabled (Perhaps via a configuration) and gauge things for your assessment, then disable it once your done. This way, you won't have to worry about the implication of using jstack or any other such approach. Actually, jstack would be Okay in this scenario. You should check this link out as well, it might help a bit (Scroll to the Description section). – Uxorial

So you are recommending approach 2 from my description above? What are your thoughts on the disadvantages I mention above? Also do you think there would be any issues in executing runtime.exec() in production environments? – Dania

On the contrary, I'm saying that jstack would do the job given your cercumstances, I'm simply suggesting you add some means in your application by which you can trigger those dumps on demand, and have a configuration to disable this feature all together as well. You don't want to keep it available on production. – Uxorial