How does jstack -F affect a running Java process?

Question

I am trying to diagnose a problem where a Java web application I'm using (Jenkins) becomes unresponsive. If I run jstack without the -F flag it doesn't give me anything, but if I put the flag in to force a thread dump not only do I get a result, but the application starts responding and goes on as if nothing had happened until it eventually stops responding again.

What does jstack -F flag do that would affect a running JVM and cause an unresponsive application to start responding again?

Grapery

Just speculation, but it might cause a spurious wake up or something similar to occur. – Querist

In order to establish what is causing the issue I would try using jdb. This way when the application is unresponsive you will be able to attach and investigate the cause. – Pathless

Answer 1

You can see the source to jstack here. The -F argument changes how jstack connects to the jvm. With -F (or -m) JStack connects to the jvm using the java debugger interface. If a pid is specified, JStack connects with the SA PID Attaching Connector which says,

The process to be debugged need not have been started in debug mode(ie, with -agentlib:jdwp or -Xrunjdwp). It is permissable for the process to be hung.

I don't know why it would cause an unresponsive application to start responding again, but the link above also says,

The process is suspended when this connector attaches and resumed when this connector detaches.

This may have an effect.

Fraley

The PID attach mechanism does not require the inspected process to collaborate - it just inspects the memory structures and tries to figure what is available. It may provide less information than if asked nicely. – Kurzawa

ok have understood why it could works better after a jstack -F. – Bane

What would change after a process is suspended and woken up that would cause the application to become responsive again? – Grapery

@JordanBentley It's hard to know why without knowing why the application is unresponsive in the first place. Is it gc'ing, swapped out, the machine itself is overloaded? – Fraley

@Fraley Most of the time when I run jmap -heap on it when it's unresponsive it will show that the eden space is full so I assume that GC is going on. I turned on concurrent GC but that didn't make any difference. The machine couldn't be overloaded, it's a powerful server with nothing else is running on it but Jenkins, and the problem still occurs when it doesn't have any builds running under it. My next step will be to try a different JVM, but I'd still like to know how jstack suspending the process would alter it's execution for my own curiosity. – Grapery

Answer 2

jstack -F -l pid is similarly to (assume working dir is JAVA_HOME)

bin/java -Dsun.jvm.hotspot.debugger.useWindbgDebugger  -Dsun.jvm.hotspot.debugger.useProcDebugger  -cp lib/sa-jdi.jar;lib/tools.jar  sun.tools.jstack.JStack -F -l pid

and in the sun.tools.jstack.JStack code

   if (arg.equals("-F")) {
       useSA = true;
   }
   .....
   // now execute using the SA JStack tool or the built-in thread dumper
   if (useSA) {
       // parameters (<pid> or <exe> <core>
       ...
       runJStackTool(mixed, locks, params);
   } else {
       // pass -l to thread dump operation to get extra lock info
       String pid = args[optionCount];
        ...
       runThreadDump(pid, params);
    }

and since -F is passed in, runJStackTool is called to load sun.jvm.hotspot.tools.JStack, it have same effect of invoking directly

bin\java -Dsun.jvm.hotspot.debugger.useWindbgDebugger  -Dsun.jvm.hotspot.debugger.useProcDebugger  -cp lib/sa-jdi.jar;lib/tools.jar  sun.jvm.hotspot.tools.JStack pid

and sun.jvm.hotspot.tools.JStack will call sun.jvm.hotspot.bugspot.BugSpotAgent attach -> go ->setupVM method

Maybe below code is the magic

       jvmdi = new ServiceabilityAgentJVMDIModule(debugger, saLibNames);
       if (jvmdi.canAttach()) {
           jvmdi.attach();
           jvmdi.setCommandTimeout(6000);
           debugPrintln("Attached to Serviceability Agent's JVMDI module.");
           // Jog VM to suspended point with JVMDI module
           resume();
           suspendJava();
           suspend();
           debugPrintln("Suspended all Java threads.");
       }

it will suspend all Java threads in the target process. if your application is hang for thread starvation, the suspend method call may relax them.

Shumate