How 'expensive' is it to execute jstack on a running JVM?

Question

I'm considering whipping up a script to

1. run once a minute (or every five minutes)
2. run jstack against a running JVM in production
3. parse the jstack output and tally up things I'm interested in
4. export the results for graphing 24/365 by a centralized Cacti installation on another server

But I've no clue as to how expensive or invasive jstack is on a running JVM. How expensive is it to execute jstack on a running JVM? Am I setting myself up for a world of hurt?

Silsby

Have you considered measuring? – Rattler

Instead of running jstack as a separate process, what about running a simple profiler within the application? It would be more efficient, because no network overhead is needed. I wrote such a tool: code.google.com/p/h2database/source/browse/trunk/h2/src/main/… - you can also convert it to a .jsp if needed. – Dom

@Thorbjørn Ravn Andersen : Pardon? – Silsby

@Thomas Mueller: Thanks, I'll have a look at your tool. But "no network overhead"? I don't think that is really a concern--I've got a fat network, and the output of any of these monitoring approaches won't add up to many bytes on the wire. – Silsby

This version is APL and doesn't have dependencies: svn.apache.org/repos/asf/jackrabbit/sandbox/jackrabbit-j3/src/… – Dom

The question is about stats collection, but the answers are about thread dumps? I presume the OP is talking about using options like -gcutil with jstack, and I am wondering the same as OP. – Issus

@Issus Nope, was not using -gcutil...I wanted the stack traces for all my running threads. – Silsby

@StuThompson Sorry, I somehow read jstack as jstat, now the answers make more sense. – Issus

Answer 1

I know this answer is late to the party but the expensive part of jstack comes from attaching to the debugger interface not generally generating the stack traces with an important exception (and the heap size does not matter at all):

Arbitrary stack traces can be generated on a safe point only or while the thread is waiting (i.e. outside java scope). If the thread is waiting/outside java scope the stack requesting thread will carry the task by doing the stack walk on its own. However, you might not wish to "interrupt" a thread to walk its own stack, esp while it is holding a lock (or doing some busy wait). Since there is no way to control the safe points - that's a risk need to be considered.

Another option compared to jstack avoiding attaching to the debugging interface: Thread.getAllStackTraces() or using the ThreadMXBean, run it in the process, save to a file and use an external tool to poll that file.

Last note: I love jstack, it's immense on production system.

Naze

@StuThompson, I am inclined to answer that as well: #3652237 the short answer is that non-direct buffers suck for real io, if you need more elaborated info, i can try... I dont post much and I tend to check profiles of people whose posts have interested me. This very question had no selected answer so I hoped I could enlighten it. – Naze

um, i was looking for something more than "non-direct bufferes suck"...that's something I've documented very well myself with benchmarks ;) I'm really curious about the discrepancies in the cliff there. – Silsby

Answer 2

Measure. One of the time variants (/usr/bin/time I believe) has a -p option which allows you to see the cpu resources used:

ravn:~ ravn$ /usr/bin/time -p echo Hello World
Hello World
real         0.32
user         0.00
sys          0.00
ravn:~ ravn$ 

This means that it took 0.32 seconds wall time, using 0.00 seconds of cpu time in user space and 0.00 seconds in kernel space.

Create a test scenario where you have a program running but not doing anything, and try comparing with the usage WITH and WITHOUT a jstack attaching e.g. every second. Then you have hard numbers and can experiment to see what would give a reasonable overhead.

My hunch would be that once every five minutes is neglectible.

Hamo

OK: real 0.54, user 0.51, sys 0.07 on an mildly loaded system. Will try later today. Besides how long it takes, what impact does jstack have on the JVM? An all stop? – Silsby

Remember to measure your application, not the jstack invocation. – Rattler

It's a web app running in Tomcat. Hmmm...maybe I should run jstack in a loop, and compare the number of invocations per minute with the CPU usage of the Java Tomcat process. Thanks suggesting this and helping think it through. – Silsby

Answer 3

Depending on the number of threads and the size of your heap, jstack could possibly be quite expensive. JStack is meant for troubleshooting and wasn't designed for stats gathering. It might be better to use some form on instrumentation or expose a JMX interface to get the information you want directly, rather than have to parse stack traces.

Caliginous

1000 threads, <1GB heap. I'm considered jstack because I know it and can implement quickly. JMX, while a viable option, is going to take some edumacation. – Silsby