Is there a way to do a live heap dump with ibm-jdk for linux?

Question

I know that it's possible to dump heap when an OutOfMemoryException is occuring on this JVM but is it possible to ask a live dump with tools like jmap or jconsole?

Tishtisha

The purpose is to find a leak that only happens in a production environment situation. I can't stop the jvm nor wait for an OutOfMemoryException to occur (too long) – Tishtisha

Answer 1

OK, I'll finally answer to myself : the application has a remote admin interface, so I will implement a new command that is calling the com.ibm.jvm.Dump.HeapDump() method.

Tishtisha

Answer 2

You need to be aware that there are "system" dumps (basically OS core files) and "heap" aka portable heap dumps (PHD). The later ones are less usefull as they do not contain actual data. They are hower enabled by default.

On AIX or Linux Typically you will set up -Xdump:system (short for -Xdump:system:events=gpf+user) to allow kill -3 <pid> to trigger a heap dump.

BTW, you can with the default options use kill -ABRT <pid>. However this will terminate your JVM.

Run java -Xdump:what to see your defaults, like:

> /usr/java6/bin/java -Xdump:what -version

Registered dump agents
----------------------
-Xdump:system:
    events=gpf+abort+traceassert,
    label=/home/u0002824/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,
    range=1..0,
    priority=999,
    request=serial
----------------------
...
java version "1.6.0"
Java(TM) SE Runtime Environment (build pap3260sr9fp2-20110627_03(SR9 FP2))

With turned on system dumps:

> /usr/java6/bin/java -Xdump:system -Xdump:what -version

Registered dump agents
----------------------
-Xdump:system:
    events=gpf+user+abort+traceassert,
    label=/home/u0002824/core.%Y%m%d.%H%M%S.%pid.%seq.dmp,
    range=1..0,
    priority=999,
    request=serial
----------------------
-Xdump:heap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=/home/u0002824/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,
    range=1..4,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD
----------------------
-Xdump:java:
    events=gpf+user+abort+traceassert,
    label=/home/u0002824/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,
    range=1..0,
    priority=400,
    request=exclusive+preempt
----------------------
-Xdump:java:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=/home/u0002824/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt,
    range=1..4,
    priority=400,
    request=exclusive+preempt
----------------------
-Xdump:snap:
    events=gpf+abort+traceassert,
    label=/home/u0002824/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,
    range=1..0,
    priority=300,
    request=serial
----------------------
-Xdump:snap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    label=/home/u0002824/Snap.%Y%m%d.%H%M%S.%pid.%seq.trc,
    range=1..4,
    priority=300,
    request=serial
----------------------
...

Do not forget to run jre/bin/jextract on the core. * .dmp files.

Megaron

Answer 3

You have a few options:

• HotSpot Diagnostic MBean
• Required settings to Generate Heapdumps
• Using the -Xdump Option

This list isn't exhaustive.

Whorish

Tried JConsole and HotSpot Diagnostic MBean on this JVM and there is no instrumentation operation available to dump heap (maybe it is available for win32 but not on linux) – Tishtisha

Regarding the previous comment, the only method available to generate live dump seems to configure dump on sigusr (events=user) using -Xdump on jvm startup. The problem with that is that uncontrolled SIGUSR may happen from time to time on the JVM... – Tishtisha

@Tishtisha - you don't say much about the product you're using. See here for some alternatives for generating system dumps: publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/… Note some of the restrictions on Linux: publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/… I would check what that does to memory consumption of a test system under load before trying in production. – Whorish

It is a big server application from our company, and we were unable to reproduce the problem under test environment. We just know that it is a supposed memory leak and that it takes a long time to grow (about 1Gb/Month) on a very very fast server with a lot of various user requests. Btw, we suspect a problem directly in IBM's classes since we already found other problems (for ex : never try to add or remove handlers to Logger while actually logging or you may get ArrayIndexOutOfBound exceptions) that SUN JDK didn't have. Btw, we may also use SUN JDK, but it is 20-30% slower... – Tishtisha

Answer 4

I think there is one tool like JProfiler . it 'll nicely work with Eclipse

Awestricken

That would suppose that I need to profile the app... Furthermore I would use the TPTP profiling tool for Eclipse. I need to dump in a live production environment. – Tishtisha