jstack not working on server

Question

We use jstack on servers to detect if java apps are getting deadlocked. It's not working on one of our Linux servers. I think O/S version is:

$cat /etc/issue.net
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Kernel \r on an \m

Java version running on server:

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

When I try:

jstack 19114

I get:

19114: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

When I try:

jstack -F 19114

I get:

Attaching to process ID 19114, please wait...
Debugger attached successfully.

Deadlock Detection:

No deadlocks found.

Thread 19180: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
        at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
        at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
        at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
        at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
        at sun.tools.jstack.JStack.main(JStack.java:84)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:51)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)

Anyone know what is causing this?

Manoeuvre

Answer 1

It is required to run jstack as the same user that started the java process. This resolves the above stacktrace error. See this posting: jstack thread dump error: get_thread_regs failed for a lwp for more detail. As soon as I sudoed the jstack command, the error disappeared.

Manoeuvre

Answer 2

The syntax is simply:

sudo -u USERID jstack PID

Example:

sudo -u tomcat7 jstack 2498

Verbalism

Yes - I use this syntax all the time now. – Manoeuvre

Answer 3

Try using kill -3 <pid> instead to get the stacktrace of your VM.

Nitriding

We do that if we want to get a vanilla thread dump. However, jstack parses the thread dump to figure out if there are any deadlocks. You can see deadlocks from vanilla thread dump, but if you have a lot of threads, it's much harder to find the problem. Jstack is much faster. – Manoeuvre

I understand your point, my suggestion is a last resort solution, that's better than being totally blind with no result from jstack. – Nitriding

Thanks, Emmanuel. There is a nice GUI tool from IBM, public.dhe.ibm.com/software/websphere/appserv/support/tools/jca/… which will parse the thread dump for you, and will identify deadlocks and other thread characteristics. We did use this approach yesterday when we couldn't get jstack to work. Thanks again for the suggestion. – Manoeuvre

linux specific i guess. but all good apps should not be on win huh ? :) – Bogie

Answer 4

Can use a jsp that gives you similar output instead.

Following jsp prints the thread stack info to screen but you could change the output to a file as well or use in a regular POJO class.

<%@ page import="java.util.*" %><%
    Map<Thread,StackTraceElement[]> map = Thread.getAllStackTraces();
    Set tt = map.keySet();
    Iterator<Thread> ti = tt.iterator();
    Thread thrd = null;
    final String br = "<" + "br" + ">";//website does not parse it
    try{
        
        int cnt = 1;
        StackTraceElement[] st = null;
        while(ti.hasNext() ){
            thrd = ti.next();
            out.print(br + "<" + "hr" + ">" + br + cnt + " \"" + thrd.getName());
            out.println("\", priority :" + thrd.getPriority()  + ", state :" + thrd.getState());
            
            out.print(", id :" + thrd.getId() + ", hex :" +  Long.toHexString(thrd.getId()) );
            out.print(" alive :"  + thrd.isAlive() + ", daemon :" + thrd.isDaemon() );
            out.print(" interrupted  :"  + thrd.isInterrupted() + ", daemon :" + thrd.isDaemon() );
            out.print(".\n" + br);
            st = thrd.getStackTrace();
            for(int sti = 0; sti < st.length; sti++){
                out.println(br + " &nbsp; &nbsp; " + st[sti].getClassName() + "." + st[sti].getMethodName());
                out.println("(" + st[sti].getFileName());
                if(st[sti].getLineNumber() < 1){
                    out.print("Native method");
                }else{
                    out.print(":" + st[sti].getLineNumber());
                }
                out.println(")");
            }
            
            out.println("");
            cnt++;
        }
    }catch(Exception e){
        out.println(br + "err " + e + br);
    }
    



%>

Sample output:

121 "Thread-40", priority :6, state :WAITING , id :134, hex :86 alive :true, daemon :false interrupted :false, daemon :false.

java.lang.Object.wait (Object.java Native method)
java.lang.Object.wait (Object.java :485)
org.jpos.iso.ISOMUX$Receiver.run (ISOMUX.java :326)
java.lang.Thread.run (Thread.java :662)

122 "Thread-48", priority :5, state :TIMED_WAITING , id :142, hex :8e alive :true, daemon :false interrupted :false, daemon :false.

java.lang.Thread.sleep (Thread.java Native method)
org.jpos.apps.qsp.QSP.monitorConfigFile (QSP.java :301)
org.jpos.apps.qsp.QSP.run (QSP.java :346)
java.lang.Thread.run (Thread.java :662)

Bogie