Huge heap dump (11GB) - Jhat failed & Eclipse MAT needed help

Question

We got a memory error in our EA and we used -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/some/dir to dump the heap at the time of OOM.

We had a heap memory of 12GB and perm gen of 256MB.

The heap dump is generated in the Linux box where we have our application running and its size is 11.5GB. We don't have privileges to download it to our local.

When we tried to analyze that 11GB heap dump using JHAT it thrown an OOM.

We tried the following commands from our Linux CLI.

jhat java_pid1491.hprof
jhat -J-Xmx16g -XX:-UseBiasedLocking java_pid1491.hprof
jhat -J-d64 -J-Xmx16g -J-XX:-UseBiasedLocking java_pid1491.hprof#1

For all the options its throwing an OOM exception after reading the dump for several minutes(>30 mins).

We googled it and found MAT as a powerful heap dump analyzer, but not a way to use it in LINUX.

Any suggestions would be of greater help. Thanks.

Modified:

Got MAT installed in Linux x86_64 machine but got following error when executed ./MemoryAnalyzer

(.:17319): GLib-GObject-WARNING **: invalid (NULL) pointer instance
(.:17319): GLib-GObject-CRITICAL **: g_signal_connect_data: assertion `G_TYPE_CHECK_INSTANCE (instance)' failed
(.:17319): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): GLib-GObject-CRITICAL **: g_object_get: assertion `G_IS_OBJECT (object)' failed
(.:17319): Gtk-WARNING **: Screen for GtkWindow not set; you must always set
a screen for a GtkWindow before using the window
(.:17319): Gdk-CRITICAL **: gdk_pango_context_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): Pango-CRITICAL **: pango_context_set_font_description: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_context_set_base_dir: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_context_set_language: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_new: assertion `context != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_text: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_attributes: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_alignment: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_ellipsize: assertion `PANGO_IS_LAYOUT (layout)' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_single_paragraph_mode: assertion `PANGO_IS_LAYOUT (layout)' failed
(.:17319): Pango-CRITICAL **: pango_layout_set_width: assertion `layout != NULL' failed
(.:17319): Pango-CRITICAL **: pango_layout_get_extents: assertion `layout != NULL' failed
(.:17319): Gtk-CRITICAL **: gtk_icon_theme_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): Gtk-CRITICAL **: gtk_settings_get_for_screen: assertion `GDK_IS_SCREEN (screen)' failed
(.:17319): Gtk-CRITICAL **: gtk_icon_size_lookup_for_settings: assertion `GTK_IS_SETTINGS (settings)' failed
(.:17319): Gtk-WARNING **: Invalid icon size 6
(.:17319): Gtk-CRITICAL **: gtk_icon_theme_load_icon: assertion `GTK_IS_ICON_THEME (icon_theme)' failed
Segmentation fault

I tried ./ParseHeapDump.sh ../java_pid1491.hprof this and got following error,

Caused by: java.lang.NoClassDefFoundError: com/ibm/dtfj/image/CorruptDataException
        at org.eclipse.mat.dtfj.InitDTFJ.stop(InitDTFJ.java:70)

Sunken

the first errors are quite normal, since there's no DISPLAY available as you are in SSH. For the second error, perhaps you could find an answer here – Terza

SAP donated MAT code to Eclipse foundation, and provide a link in case of lack of dtfj. Maybe if you install it... – Terza

I already checked first link you gave. I replaced the given jars with the feature jars available. It made entire MAT corrupted. Second link, I am yet to test and will get back to you. Thanks. – Sunken

Answer 1

I was using MAT on Linux yesterday.... check out the proper version here

Terza

@Terza - I'm downloading the file.. is there any docs for setting up that in linux.. pls share if you have any. thanks. – Sunken

that's absolutely straightforward... just cd the install directory and execute ./MemoryAnalyzer. Then go to File and parse [or load] heap dump – Terza

I couldn't execute using ./MemoryAnalyzer, but I used ./ParseHeapDump.sh ../java_pid1491.hprof to parse the file. Everything works as expected (parsing, scanning, extracting, Re-indexing, Writing), but after a while it throws java.lang.NoClassDefFoundError: com/ibm/dtfj/image/CorruptDataException. I couldn't find any solution googling it. Why it is asking for IBM's tool class when I am trying to run it for my Sun JVM. Pls help me. – Sunken

what happens when you try to execute MemoryAnalyzer ? MAT is a follow-up of an IBM exclusive tool. So, I'm not surprised to see IBM classes in it... Which Linux are you running ? – Terza

I'm running it in Linux x86_64 using SSH. Error occurred related to GUI settings. If you want full error log, I can post that too. Any clues? – Sunken

Can't you have X forwarding in SSH to run MAT anyway ? And yes, please post a more consequent stack trace if you can (you can edit your first post, maybe) – Terza

I have edited the original question and added the exception what i got. Thnks 4 ur help. – Sunken

MAT is not an follow-up of an IBM exclusive tool. It's a followup of a SAP tool. – Tern

@kolherm I've seen this while searching information about it, but didn't correct myself in the answer, but in comments to the questions. Thanks for spotting – Terza

Answer 2

forget Jhat, you would need more than 100Gybte of RAM to parse this dump, it would take forever and it would be very hard to find something. I just lately installed MAT on a SUSE 11 machine. worked flawlessly . Can you please make sure that MAT is using the right JVM? Not sure whether we testet it with anything else then the SUN/SAP JVMs.

Regards, Markus (kohlerm)

Tern

Answer 3

We have had success in the past using Netbeans to analyze OOM errors. It seemed to cope where Eclipse and JHAT couldn't.

Henequen