jstack: Target process not responding

Question

I am running Ubuntu server edition and I wanted to take a thread dump of Tomcat.

So, I first tried to find out which PID tomcat uses:

$ jps -l
5809 sun.tools.jps.Jps

But it's not there?

So, I used top instead and found out the PID 5730.

Then I called jstack to get the thread dump:

$ sudo jstack -l 5730
5730: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

What's going on? :-(

I already tried to export CATALINA_TMPDIR as described in Jstack and Jstat stopped working with upgrade to JDK6u23 but that didn't change anything:

$ export CATALINA_TMPDIR=/tmp
$ sudo /etc/init.d/tomcat6 restart
 * Stopping Tomcat servlet engine tomcat6
   ...done.
 * Starting Tomcat servlet engine tomcat6
   ...done.
$ sudo jstack -l 5934 // new PID after restart
5934: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Update:

I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console.

Nine

Have you tried the -F option? – Echolalia

@Michael If I use the -F option, then it actually starts the thread dump but the dump itself just contains a lot of error messages: dl.dropbox.com/u/17844821/zeug/threaddump.txt and during the dumping process many many exceptions are thrown: dl.dropbox.com/u/17844821/zeug/threaddumpexception.txt – Nine

@valmar Is that possible that the jdk used by Tomcat and jdk you are using for jps are not the same jdk? Could you double check it? – Kappenne

@Clark Mh, how would I do that? – Nine

@valmar check this. howtogeek.com/howto/linux/installing-tomcat-6-on-ubuntu Tomcat relys on JAVA_HOME. but your command line, you can type java -version to see if they are the same version and check the java bin path the command are using. – Kappenne

@Clark Thanks for your reply! I installed Tomcat simply via sudo apt-get install tomcat6 (no manual installation via wget). echo $JAVA_HOME returns me nothing! (Empty line) java -version gives me this: java version "1.6.0_20" OpenJDK Runtime Environment (IcedTea6 1.9.9) (6b20-1.9.9-0ubuntu1~10.04.2) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode) Does this make any sense to you? – Nine

@valmar Can you try export JAVA_HOME=<path of OpenJDK 64-Bit> and then restart the Tomcat and then try the jps again? – Kappenne

@ClarkBao Nope, that didn't do the trick. jps -l still returns only 2316 sun.tools.jps.Jps. echo $JAVA_HOME gives /usr/lib/jvm/java-6-openjdk – Nine

@valmar I cannot look into the script of how Tomcat6 start the java process. You may need to debug into it to see which jvm it is really using. – Kappenne

@ClarkBao I checked the script and the JDK path seems to be correct or am I missing something? dl.dropbox.com/u/17844821/zeug/tomcat_script.txt – Nine

@valmar I cannot open the link. But there is another way to get thread dump. Send server a SIGQUIT signal to force a stack dump.Check this. #7136182 – Kappenne

@ClarkBao Is stack dump = thread dump? Because your link only describes how to get a stack dump. Here is a new link to the script btw: zeug.grandia2.de/tomcat_script.txt – Nine

@ClarkBao I just tried sudo kill -QUIT 5244 but that returns me nothing (just a blank new command line) and nothing happens (5244 is the PID of tomcat and -QUIT should send a SIGQUIT to that process). – Nine

Answer 1

I got it working by doing two things:

1. Changed call to: sudo -u tomcat6 jstack -J-d64 -m pid
2. Replaced OpenJDK with Sun's original sun-6-jdk and sun-6-jre packages

---

Explanation for part 1: I switched to 64-bit mode, used sudo and run the command as Tomcat user.

Note: Part 2 might not be necessary. For some users it seems like part 1 is enough. In fact, try to add just the sudo command first. It might already do the trick.

Nine

Placing sudo -u <javaProcUsername> in front of the command did the trick for me. – Kironde

By the way - just by running as the same user, I find I eliminate this error - in my case attempting to connect with yourkit, but I believe the same principle applies. Being in Sun JDK vs. OpenJDK, or specifying 64 vs. 32 bit seems unnecessary for me to fix it. – Catarrhine

Running the jstack command with the same user under which the jvm was running fixed the issue for me. Thanks @DavidW ! – Calif

The -J-d64 from root did the trick for me, I tried every combination, correct user, sudo, root. With and without native frames (-m) but it started working when I gave the -J-d64 flag, I also needed -F to get around the process being unresponsive. – Gujarat

After switching to tomcat's user, the -F flag was actually making it fail for me. – Spoliation

Answer 2

I think you need to run jstack as the same user that runs the Tomcat process. Note also that jps only returns processes for the current user. You would get the pid for the Tomcat process by running jps with sudo or as the Tomcat process user.

This bug report may also be useful: https://bugs.launchpad.net/ubuntu/+source/sun-java6/+bug/597098

Echolalia

I can't switch to tomcat6 user because it's not actually a "real" unix account on the system. So su tomcat6 doesn't work. I just tried sudo jps -l but it still only gives me 6493 sun.tools.jps.Jps – Nine

I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console: dl.dropbox.com/u/17844821/zeug/threaddump2.txt - The threaddump itself looks like this: dl.dropbox.com/u/17844821/zeug/threaddumpexceptions2.txt - Doesn't look right to me. – Nine

@valmar: try this to become tomcat6: switches to root: sudo su - from root you you can switch to any user su tomcat6 – Offend

@Cd-MaN No effect. If I do that, it gives me this (the slashes represents newlines since I cannot post newline characters here in comments): $ whoami // valmar // $ sudo su - // # whoami // root // # su tomcat6 // # whoami // root – Nine

@valmar: this probably means that you do't have a shell set for the given user (tomcat6) - or most precisely you have the shell set to something like /bin/false. To solve this, use su -p tomcat6 from under the root account (this preserves the current shell). – Offend

Answer 3

@Valmar, I find the same topic post here. Unable to get thread dump? Any ideas why my app blocks?

It seems the workaround is sudo -u tomcat6 kill -3 <pid>.

Kappenne

Nop, that didn't do the trick. When I do that, nothing happens. – Nine

if you are using 64bit, you need to use jstack -J-d64 -m pid, also try to add sudu -u tomcat6 . download.oracle.com/javase/6/docs/technotes/tools/share/… – Kappenne

Hm, that only gives me a load of exceptions on the console: dl.dropbox.com/u/17844821/zeug/console64.txt – Nine

If I were you,I will try to use another jdk not openjdk. Maybe it's due to openjdk. – Kappenne

Interesting. I replaced openjdk-6-jre with sun-java6-jre and now jstack seems to work properly. jps still doesn't show tomcat and its PID though. @Clark Thanks for your help so far! If you post your suggestion to replace the JRE as a new, separate answer, I'll mark it as "the right answer". – Nine

Answer 4

Try to switch to process user and then use jstack:

sudo -u {process user} jstack > dump

Barrybarrymore

Answer 5

This also worked for me:

sudo -u tomcat6 kill -3 pid

It looks like nothing happens but when you look in the logs the stacks are there. They look like exceptions if your not expecting them.

Shields

Answer 6

I find it useful to use something like 'ps -eo pid,user,command | grep java' to find the actual java command being used, then use the directory to find the matching jstack etc.

# ps -eo user,command | grep '[j]ava' | cut -d' ' -f1
someuser /usr/lib/jvm/java/bin/java

# /usr/lib/jvm/java/bin/java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

So its 64-bit, running as 'someuser'. su to that user and run run jstack etc. from that same directory. (ie. /usr/lib/jvm/java/bin/jstack

Useful when you're on a server with various different installations / implementations of Java.

Organotherapy

Answer 7

For Tomcat users having this issue, check your catalina.out log file.

I was having the same problems "22693: Unable to open socket file: target process not responding or HotSpot VM not loaded". I gave up and was trying to find anything about what happened before it locked up, but then there was the jstack output in the log file.

Eternal

Answer 8

I had same problem, but none of below solution worked for me:

jstack <pid>
jstack -J-d64 -m <pid>
sudo -u <user> jstack ...

I finally upgraded JDK from jdk1.6.0_24 to jdk1.7.0_67 and every things worked.

Cognizance

Answer 9

For those finding this answer six years after it was asked and they are on CentOS 7 note that SELinux will stop jmap from writing the heap dumps as, out of the box, it will deny writing to a socket unless the directory has the type tomcat_tmp_t.

In my case, I write my heap dumps under /usr/share/tomcat/.jmap. This directory is owned by my runtime user.

Therefore to allow jmap to write to this directory:

semanage fcontext -a -t tomcat_tmp_t "/usr/share/tomcat/.jmap(/.*)?"
restorecon /usr/share/tomcat -vR

Which allows us to then run, as our tomcat user:

jmap -dump:format=b,file=/usr/share/tomcat/.jmap/tomcat-`date +%s`.bin <pid>

Rb

Answer 10

My problem was, I ran the process by the terminal. Then for testing Its jstack, I paused the process by ctrl+z. The process was not able to respond. To solve this problem, I resumed the process by fg and checked its jstack by another terminal.

Typographer