Jmap can't connect to make a dump

Question

We have an open beta of an app which occasionally causes the heapspace to overflow. The JVM reacts by going on a permanent vacation.

To analyze this I would like to peek into the memory at the point where it failed. Java does not want me to do this. The process is still in memory but it doesn't seem to be recognized as a java process.

The server in question is a debian Lenny server, Java 6u14

/opt/jdk/bin# ./jmap -F -dump:format=b,file=/tmp/apidump.hprof 11175
Attaching to process ID 11175, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypeEntryTypeNameOffset" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:390)
at sun.jvm.hotspot.HotSpotTypeDataBase.getLongValueFromProcess(HotSpotTypeDataBase.java:371)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:102)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:568)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Debugger attached successfully.
sun.jvm.hotspot.tools.HeapDumper requires a java VM process/core!

Brianbriana

I see the jmap is executed from the Hotspot JVM you have installed. Are you sure the application itself was launched with the same Java version? – Latakia

I rechecked just to be sure, but there is only one version of java available on the machine and I call jmap directly from the bin dir. No PATH errors. So it is definately the same JVM. A possibility someone here offered was that it has to do with a -XX:+UseCompressedOops parameter. – Brianbriana

I had gotten an error like this: "Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process – Winsor

Under Linux you may need "sudo ..." to get proper access to the process. – Aim

Answer 1

The solution was very simple. I was running the jmap as root, but I had to run it as the user who started the jvm. I will now go hide my head in shame.

Brianbriana

OR , Sun can make the error less cryptic instead of saying "requires a java VM process/core!" – Eyecup

Hello, what command did you exactly used to solve this? I'm trying with su -m myuser -c 'jmap -J-d64 -dump:format=b,file=/tmp/apidump.hprof 98233', but all I am getting is Operation not permitted as a response. – Marabou

./jmap -F -dump:format=b,file=/tmp/apidump.hprof 11175 but this was all a long time ago – Brianbriana

Nine years later, still helping people. :) – Hypodermis

Open JDK 14 ` jhsdb jmap --heap --pid 1` – Silvertongued

Answer 2

I was running the jmap and the application with the same user and still get the error.

The solution was run that comand before the jmap

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Than is just use jmap and will works fine

jmap -heap 17210

Rurik

Documented here: "The error is non obvious, but comes from a configuration of ptrace in Ubuntu." This command reconfigures. Logged as JDK-7050524 – Pulitzer

the document link provided by Andy is broken. This helps: askubuntu.com/questions/146160/… – Bookkeeper

It also happened to me on Ubuntu 18 and OpenJDK 11.0.2. jhsdb jmap --pid 8236 Attaching to process ID 8236, please wait... Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 8236: Operation not permitted – Secularity

Answer 3

If someone tries to get Heap Dump of Java application in Docker container. This is the only solution that worked for me:

docker exec <container-name> jcmd 1 GC.heap_dump /tmp/docker.hprof

It basically dumps the heap of process with pid=1 using jcmd

See https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html

Bonnette

Your link seems to be dead. Can you please share the proper link? – Imbue

Links works for me now, text was updated after @Imbue comment. – Gratian

This did not work for me, when the java process running in Docker is running as a non-root user. When running as root, I get Unable to open socket file: target process not responding or HotSpot VM not loaded, when running as the non-root user I get Permission denied. – Gratian

Answer 4

Future Googlers:

This could also happen if you installed the JDK while the process you're trying to jmap was running.

If that's the case, restart the java process.

Caboodle

What if you can't restart the java process because you want to investigate the system in the current state? That's my case right now :( . I did precisely that, I installed the JDK while the process was running. – Pointer

Answer 5

Follow the below steps to take the thread and Heap dumps from a docker container

1. Run the below command to bash into the container. Please change the CONTAINER_NAME appropriately

   docker exec -it CONTAINER_NAME bash

2. Then type jps to find the all the Java application details and extract the PID for your application

jps

3. Then run the below command to get the thread dump. Please change the PID appropriately

   jstack PID > threadDump.tdump 
   

4. Then run the below command to get the Heap dump. Please change the PID appropriately

    jmap -dump:live,format=b,file=heapDump.hprof PID 

5. Then exit from the docker container and download the threadDump.tdump and heapDump.hprof from the docker container by running the below command. Please change the CONTAINER_NAME appropriately

 sudo docker cp CONTAINER_NAME:threadDump.tdump .
 sudo docker cp CONTAINER_NAME:heapDump.hprof .

Zillah

1.Execute "Docker ps", will give the container Id of all services and collect the container id foe TSC. 2.Execute "docker exec -it CONTAINER_ID bash" (replace CONTAINER_ID with TSC Container id) 3.Bash will come and then execute the "jps" on bash, that will give you the PID for process(it will be 1 for jar) 4.Execute the "jstack PID > threadDump.tdump"(replace PID with process id received in step 3, it should be 1) 5.Execute the "jmap -dump:format=b,file=heapDump.hprof PID"(replace PID with process id received in step 3, it should be 1) – Proceleusmatic

Answer 6

What happens if you just run

./jmap -heap 11175 

And are you sure the application JVM is identical to the JMAP JVM? (same version, etc)

Airplane

In my case: ` /opt/jvm/jdk1.7.0_15/bin/jmap -heap 2022 Attaching to process ID 2022, please wait... Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process` – Venepuncture

Answer 7

You need to use the jmap that comes with the JVM.

Paymar

Answer 8

I got the same jmap error on a linux machine that have two different OpenJdks installed. First I installed OpenJDK 1.6 and after that OpenJDK 1.7.

A call of ...

/usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java -XshowSettings:properties -version

# produce the following output ...
...
java.library.path = /usr/java/packages/lib/amd64
    /usr/lib/x86_64-linux-gnu/jni
    /lib/x86_64-linux-gnu
    /usr/lib/x86_64-linux-gnu
    /usr/lib/jni
    /lib
    /usr/lib
...
java version "1.7.0_65"

With including '/usr/lib' every with OpenJDK 1.7.* started program includes the libraries of the first installed JDK (in my case OpenJDK 1.6.*). So the jmap versions of Java6 and Java7 failed.

After I changed the start for the Java7 programms with included OpenJDK 1.7 libraries ...

/usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java -Djava.library.path=/usr/lib/jvm/java- \
                  7-openjdk-amd64/jre/lib/amd64/server:/usr/java/packages/lib/amd64: \
                  /usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/ \
                  x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib ...

I was able access proccess with the Java 7 version of the jmap program. But it needs a sudo to run.

Fossa

Answer 9

I have the same problem, I'm trying to find a memory leak in a process running inside a Docker container. I wasn't able to use jmap, instead I used this:

jcmd <pid> GC.class_histogram 

This gives you a list of the objects in the memory. And from the Oracle documentation:

It is recommended to use the latest utility, jcmd instead of jmap utility for enhanced diagnostics and reduced performance overhead. https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks004.html

Substitutive

Answer 10

1.Execute "Docker ps", will give the container Id of all services and collect the container id foe TSC.
2.Execute "docker exec -it CONTAINER_ID bash" (replace CONTAINER_ID with TSC Container id)
3.Bash will come and then execute the "jps" on bash, that will give you the PID for process(it will be 1 for jar)
4.Execute the "jstack PID > threadDump.tdump"(replace PID with process id received in step 3, it should be 1)
5.Execute the "jmap -dump:format=b,file=heapDump.hprof PID"(replace PID with process id received in step 3, it should be 1)
6.Then we have to exit the bash using "exit" command
7.Execute "sudo docker cp CONTAINER_ID:heapDump.hprof ." from ec2 command line, that will copy the dump file on ec2 machine present working directory.
8.Execute "sudo docker cp CONTAINER_ID:threadDump.tdump ." from ec2 command line, that will copy the dump file on ec2 machine present working directory.

Proceleusmatic

Answer 11

When none of these work or if you don't want to change sensitive OS flags such as ptrace_scope:

Either you can use jconsole/jvisualvm to trigger heap dumps or run any JMX client directly from console as follows as you are doing it locally on the machine that needs the dump and so is faster:

echo 'jmx_invoke -m com.sun.management:type=HotSpotDiagnostic dumpHeap heapdump-20160309.hprof false' | java -jar jmxsh.jar -h $LOCALHOST_OR_IP -p $JMX_PORT

I used the wget https://github.com/davr/jmxsh/raw/master/jmxsh.jar for this example.

Arkhangelsk

Answer 12

What worked for me was to simply issue the command with sudo as in:

sudo jmap -heap 21797

Cindicindie

Answer 13

In my case it is not as simple as check the user :(

I have a script called collectd-java which invokes jstat and jmap. I've checked by top that such script is launched, as expected, by the user owning the JVM. However, jstat gives me what I need and jmap can't attach. Here is the script - the echo stuff is just the format I need to present the values:

HOSTNAME="${COLLECTD_HOSTNAME:-localhost}"
INTERVAL="${COLLECTD_INTERVAL:-60}"
MAIN_CLASS="my.fully.qualified.MainClass"
PID=$(pgrep -f ${MAIN_CLASS})

get_jstat_classloaderdata() {
VALUE=`jstat -class $PID 1 1 | awk '{print $1}' | grep -vi loaded`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_loaded\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $2}' | grep -vi bytes`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_bytesload\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $3}' | grep -vi unload`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_unloaded\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $4}' | grep -vi bytes`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_bytesunload\" interval=$INTERVAL N:$VALUE"

VALUE=`jstat -class $PID 1 1 | awk '{print $5}' | grep -vi time`
echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-java_classloader_time\" interval=$INTERVAL N:$VALUE"
}

get_jmap_heapdata() {
        VALUE=$(jmap -heap ${PID} | grep MinHeapFreeRatio |awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_minheapfreeratio\" interval=$INTERVAL N:$VALUE"

        VALUE=$(jmap -heap ${PID} | grep   MaxHeapFreeRatio|awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_maxheapfreeratio\" interval=$INTERVAL N:$VALUE"

        VALUE=$(jmap -heap ${PID} | grep   MaxHeapSize|awk '{print $3}')
        echo "PUTVAL \"$HOSTNAME/exec-cecoco/gauge-jmap_maxheapsize\" interval=$INTERVAL N:$VALUE"
}
##Do it
get_jmap_heapdata
get_jstat_classloaderdata

Jstat succeeds and jmap fails. Does anyone understands it ?

Protamine

Sorry, I forgot. Such script it launched from application Collectd, which ensures the right user (as I said). But if I execute directly as root such script, it all works right !!! WTF :( :( !?!?!? – Protamine

Answer 14

Not sure why a plain "jmap " fails when I docker exec -it into my container running centos7 systemd and a java service, but below jmap options worked for me. Thanks: https://dkbalachandar.wordpress.com/2016/07/05/thread-dump-from-a-docker-container/

[root@b29924306cfe /]# jmap 170 Attaching to process ID 170, please wait... Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 170: Operation not permitted sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 170: Operation not permitted

[root@b29924306cfe /]# jmap -dump:live,format=b,file=heapDump.hprof 170 Dumping heap to /heapDump.hprof ... Heap dump file created

Gisborne