Garbage Collector First and JMap EOF bug

Question

We are working over our client's production server heap to detect and solve memory leaks. For this we are using jmap periodically to collect the necessary information.

But last week we couldn't take the dump, because it triggered a EOF error and shutdown the Tomcat instance.

I searched on the internet but couldn't find any concrete information about this error. We detected that it only occurs when using the Gc First garbage collector algorithm.

This is the command line we used to perform the jmap:

jmap -dump:format=b,file=heap.bin <PID>

Java version on the server: JDK 1.7.0_7 x64

Has anyone already faced this kind of error? Maybe some configuration that is missing or a patch to java/jmap required.

UPDATE

A few more information that we have collected about this error:

[root]# jmap -dump:format=b,file=heap.bin 7806
    Dumping heap to /tmp/heap.bin ...
    Exception in thread "main" java.io.IOException: Premature EOF
        at sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:244)
        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:193)
        at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:213)
        at sun.tools.attach.HotSpotVirtualMachine.dumpHeap(HotSpotVirtualMachine.java:180)
        at sun.tools.jmap.JMap.dump(JMap.java:241)
        at sun.tools.jmap.JMap.main(JMap.java:140)
[root]#

Note: the target directory has over 500gb of free space

Error outputed to the catalina.out (JVM dump error):

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f0269cc41c6, pid=7806, tid=139647231129360
#
# JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build 1.7.0_40-b43)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b56 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x58c1c6]  DumperSupport::dump_field_value(DumpWriter*, char, unsigned char*)+0x1c6
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/tomcat6/bin/hs_err_pid7806.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp

Winonawinonah

I am just facing the same problem with JDK 1.7.0_45. Most heap dumps crash the VM an dump a core with the same problematic frame that you mentioned. This makes analysing memory leaks in java impossible. Did you find any solution to the problem? The only thing I can think of now is to switch back to CMS / other GC. I reported it with Bug ID: 900949 – Milinda

I still don't have any solution to these problem @mnp. I've also reported this in december with Bug ID 9008971, without any response yet. – Winonawinonah

Hello, I can't find either of your two bugs in the oracle bug db, the only related bug I could see is resolved bugs.java.com/bugdatabase/view_bug.do?bug_id=6882554 I can however confirm that this is still happening on 1.7u45 with G1 – Acidic

Hello @DrGranit! I'm also not able to find the bug report in the oracle bug db. It's really strange, but I have followed all the instructions on the JVM dump and fulfilled the bug report submission. I really don't know where these information goes, I just hope that someboy reads it :) – Winonawinonah

Still happening with jdk 1.7u55. – Milinda

Maybe we should open a new case? – Lamee

This still happens with JDK 1.7u65. Using option "-F" the dump can be generated successfully, but it takes forever so that's not really a solution for larger heaps. – Shelleyshellfire

Any updates on that? The issue still persists in java 1.8.0_45 – Billhook

It sounds like you should all file bugs against the JDK. – Stratocumulus

I'm voting to close this question as off-topic because it's a Java bug report, not a question. – Stratocumulus

@Stratocumulus a bug report was filed at the time. When I made the question the idea was to know if the problem only happend with me or if it happened because of some mistake I made. I agree that at its point its probably a real bug and it does not fit in the Stackoverflow format. – Winonawinonah

@Winonawinonah do you have a link to the bug you filed? It would be helpful for other people who find this, I'm sure. – Stratocumulus

@Stratocumulus when I submited I recived the ID 9008971 in a e-mail. But I can find it on the bug database at bugs.sun.com. This email was send to me on 12-16-2013 – Winonawinonah

@Winonawinonah according to the FAQ "Not all bugs submitted do end up in the database." perhaps they didn't have enough information from your initial report? If it's repeatable, an SSCCE might help them. – Stratocumulus

Is there any resolution to this issue? – Cryptonymous

Answer 1

I could able to resolve the issue by using a few additional options.

java version "1.7.0_45".

Java process was configured with first garbage collection algorithm: -XX:+UseG1GC

jmap -J-d64 -dump:live,format=b,file=<heap_dump_filename> <PID>

Cryptonymous

Please let me know the exact command to add java version in jmap command. I did not find that option in internet. I have same issue @ #36604602 – Luckey

You don't need to pass java version as a parameter with jmap command. You can find java version using java -version command though!! – Cryptonymous