Core dump taken with gcore, jmap conversion to hprof file format fails with Error message

Question

We recently had one of our JVM's crash, leaving behind a core dump file produced by the gcore command. We want to have a look at the contents of the file and find out exactly what caused the crash.

Using the jmap command you are supposed to be able to turn core dump files into files in the hprof file format, which you can then analyse using VisualVM and a number of other tools. I've tried this and get an error message. This was the command that I ran (on the same box that the crash took place, using the same JVM):

jmap -dump:format=b,file=dump.hprof /usr/java/jdk1.6.0_16/bin/java core.dump.2878

The response in it's entirety was:

> Attaching to core core.dump.8483 from executable /usr/java/jdk1.6.0_16/bin/java, please wait...
> Error attaching to core file: Can't attach to the core file

That's not a very helpful error message. I wondered if it was a permissions issue, but I get the same message running the command as the same use that ran the JVM that caused the core dump. I also wondered if the core file was corrupt, so I decided to use gdb to see if I could open up the core file and see what was in it. This is what I get:

> gdb
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) core-file core.dump.8483
[New Thread 2889]
[New Thread 2893]
[New Thread 2894]
[New Thread 2895]
[New Thread 2896]
[New Thread 2904]
[New Thread 2915]
[New Thread 2916]
[New Thread 2917]
[New Thread 2921]
[New Thread 2922]
[New Thread 3175]
[New Thread 3239]
[New Thread 3252]
[New Thread 3258]
[New Thread 3260]
[New Thread 3356]
[New Thread 3509]
[New Thread 3510]
[New Thread 3514]
[New Thread 3523]
[New Thread 3541]
[New Thread 3542]
[New Thread 3543]
[New Thread 4022]
[New Thread 4057]
[New Thread 4058]
[New Thread 4077]
[New Thread 4078]
[New Thread 4079]
[New Thread 4080]
[New Thread 6128]
[New Thread 6140]
[New Thread 6162]
[New Thread 6376]
[New Thread 6389]
[New Thread 6408]
[New Thread 6422]
[New Thread 6429]
[New Thread 6451]
[New Thread 6497]
[New Thread 6513]
[New Thread 6514]
[New Thread 6516]
[New Thread 6517]
[New Thread 6532]
[New Thread 6533]
[New Thread 6665]
[New Thread 6675]
[New Thread 6676]
[New Thread 6687]
[New Thread 6689]
[New Thread 6692]
[New Thread 6706]
[New Thread 6707]
[New Thread 6735]
[New Thread 6736]
[New Thread 7033]
[New Thread 7034]
[New Thread 7056]
[New Thread 7077]
[New Thread 7079]
[New Thread 7080]
[New Thread 7082]
[New Thread 7089]
[New Thread 7090]
[New Thread 7091]
[New Thread 7092]
[New Thread 7103]
[New Thread 7105]
[New Thread 7107]
[New Thread 7108]
[New Thread 7116]
[New Thread 7229]
[New Thread 7308]
[New Thread 7493]
[New Thread 7505]
[New Thread 7510]
[New Thread 7511]
[New Thread 7517]
[New Thread 7523]
[New Thread 7604]
[New Thread 7617]
[New Thread 7618]
[New Thread 7619]
[New Thread 8676]
[New Thread 8693]
[New Thread 8700]
[New Thread 8851]
[New Thread 8860]
[New Thread 8887]
[New Thread 9007]
[New Thread 9118]
[New Thread 9119]
[New Thread 9120]
[New Thread 9413]
[New Thread 9427]
[New Thread 9495]
[New Thread 9508]
[New Thread 9519]
[New Thread 9535]
[New Thread 9536]
[New Thread 9537]
[New Thread 9554]
[New Thread 9556]
[New Thread 9659]
[New Thread 9660]
[New Thread 9663]
[New Thread 9664]
[New Thread 9665]
[New Thread 9666]
[New Thread 9667]
[New Thread 9668]
[New Thread 9669]
[New Thread 9670]
[New Thread 9671]
[New Thread 9678]
[New Thread 9870]
[New Thread 9953]
[New Thread 9998]
[New Thread 10002]
[New Thread 10118]
[New Thread 10119]
[New Thread 10122]
[New Thread 10149]
[New Thread 10152]
[New Thread 10155]
[New Thread 10176]
[New Thread 10178]
[New Thread 10179]
[New Thread 10180]
[New Thread 10182]
[New Thread 10194]
[New Thread 10195]
[New Thread 10196]
[New Thread 10198]
[New Thread 10199]
[New Thread 10200]
[New Thread 10201]
[New Thread 10202]
[New Thread 10203]
[New Thread 10205]
[New Thread 10206]
[New Thread 10244]
[New Thread 10246]
[New Thread 10247]
[New Thread 10248]
[New Thread 10249]
[New Thread 10251]
[New Thread 10252]
[New Thread 10254]
[New Thread 10255]
[New Thread 10256]
[New Thread 10257]
[New Thread 10258]
[New Thread 10259]
[New Thread 10260]
[New Thread 10261]
[New Thread 10262]
[New Thread 10263]
[New Thread 10264]
[New Thread 10265]
[New Thread 10267]
[New Thread 10268]
[New Thread 10269]
[New Thread 10271]
[New Thread 10476]
[New Thread 10477]
[New Thread 10479]
[New Thread 10552]
[New Thread 10607]
[New Thread 10611]
[New Thread 10612]
[New Thread 10613]
[New Thread 10615]
[New Thread 10617]
[New Thread 10623]
[New Thread 10624]
[New Thread 10625]
[New Thread 10641]
[New Thread 10642]
[New Thread 10649]
[New Thread 10736]
[New Thread 10742]
[New Thread 10756]
[New Thread 10758]
[New Thread 10760]
[New Thread 10761]
[New Thread 10762]
[New Thread 11278]
[New Thread 11412]
[New Thread 11513]
[New Thread 11514]
[New Thread 2878]
(gdb) quit

And at that point I quit, because I know absolutely nothing about gbd and how to use it to diagnose this sort of issue. I don't even really understand what that last command did. One thing worth noting is that there are exactly 134 of those "New Thread" lines present in the output, and, if each one represents a new thread spawning in the JVM, this could be the reason the JVM died.

So my question is in fact, three fold -

1) Any idea why the jmap command may have given that error message?

2) any ideas what the gdb output means?

3) any idea how to use gdb to further diagnose this issue?

Charwoman

You said you're running with the same user, but is it the exact same version of the jvm you're specifying? Long shot... – Referential

Yes, it is. The path was grabbed from running ps -ef and seeing the JVM the app was using. – Charwoman

Answer 1

By the way, jvisualvm can load core dumps directly. But you must use the same jvm that created the core file.

Referential

I don't see an option anywhere in VisualVM that allows you to load a core dump. I see options for loading ".apps", ".tdump", ".hprof", ".nps", ".npss". None of those selected options causes the core dump file to become visible within the "select a file" dialogue. – Charwoman

@Charwoman Go to File -> Add VM Coredump. See here for more info: docs.oracle.com/javase/6/docs/technotes/guides/visualvm/… – Referential

Okay, read the docs and it looks like that should work, except that isn't available on my Windows desktop (it's *nix only), and the target server that produced the core dump has no X window system running. – Charwoman

It is easily possible to run X Server on windows and target linux DISPLAY to it: #40953. VisualVM would run on remote linux but appear as local window. – Basifixed

You need to use the same java, so if you have a linux/unix core file you cant of course open it with Windows jvisualvm. Install a VM with linux to get a gui or use jmap to extract a hprof file. – Fishery

Answer 2

Was the core file larger than 2GB? If so, you could be having an issue with the Linux build of libsaproc.so that comes with the JVM.

Run your command again, but like this:

strace -o out.txt -f $yourOriginalCommand

Then 'grep core.2878 out.txt' and look for an error on the open() syscall. Did it return an error (E_XXXXX) or a file handle number?

Grantgranta

It's 1.8G according to du -h core.dump.2878. The distro on the remote server doesn't seem to have an strace command available (Red Hat Linux). – Charwoman

Jon - I had to download openjdk6 source and rebuild libsaproc.so with large file support to use the JDK tools. You're close to 2GB. If your core file size, in bytes, is larger than 2^31, you could have the same issue. If you have an openjdk buld environment, edit openjdk-6-src-b22-28_feb_2011/hotspot/make/linux/makefiles/saproc.make to include -D_FILE_OFFSET_BITS=64 for the $(LIBSAPROC) section. Then LD_PRELOAD=your/libsaproc.so jmap ... – Grantgranta

Answer 3

This was bothering the heck out of me as I had a core file that represented a heap that I needed to analyze, but I was constantly seeing the exception message below:

sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypeEntryTypeNameOffset" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)

Copying the jre from my source machine (the machine where the core file was obtained) on to the exact same folder in the destination machine, and then running jmap with that java location as an argument worked for me.

So here are the steps to try in case someone else runs into this:
1. Connect to the core file through gdb and confirm the location of java binary which the running process was using:

    gdb --core=</path/to/core-file>

2. The above output will end with something like

[New Thread 22748]
**Core was generated by `/opt/blah/location/jre/bin/java -Xmx...'.**

3. Make sure you copy the matching version of the jre into the /opt/blah/location/ directory

4. Then launch jmap as:

   /opt/jdk1.8.0_09/bin/jmap -heap /opt/blah/location/jre/bin/java /path/to/core-file
   

   This should connect to the core file successfully and print out heap statistics. If it does, then you have successfully read the core file

5. From that point on, you can generate the hprof from the core file successfully using:

   /opt/jdk1.8.0_09/bin/jmap -dump:format=b,file=my-file.hprof /opt/blah/location/jre/bin/java /path/to/core-file
   

Pod

It should be /opt/jdk1.8.0_09/bin/jmap -dump:format=b,file=my-file.hprof /opt/blah/location/jre/bin/java /path/to/core-file – Sudanic