how to generate thread dump java on out of memory error

Question

• does java 6 generate thread dump in addition to heap dump (java_pid14941.hprof)

• this is what happened to one of my applications.

  java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid14941.hprof ...

• I did find ava_pid14941.hprof in working directory, but didn't find any file which contains thread dump. I need to know what all the threads were doing when I got this OutOfMemory error.

• Is there any configuration option which will generate thread dump in addition to heap dump on out of memory exception?

Twink

Answer 1

How to generate thread dump java on out of memory error?

Your question can be simplified into:

• how to generate a thread dump

and:

• how to catch an out of memory error (don't pay attention to naysayer here, they're missing the bigger picture, see my comment)

So it's actually quite easy, you could do it like this:

• install a default uncaught exception handler

• upon catching an uncaught exception, check if you have an OutOfMemoryError

• if you have an OutOfMemoryError, generate yourself a full thread dump and either ask the user to send it to you by email or offer to send it automatically

Bonus: it works fine on 1.5 too :)

 Thread.setDefaultUncaughtExceptionHandler( new Thread.UncaughtExceptionHandler() {
     public void uncaughtException( final Thread t, final Throwable e ) {
         ...
     }

You may want to look into this:

 e.getMessage();

and this:

Thread.getAllStackTraces();

I'm doing this all the time in an app that is shipped on hundreds of different 1.5 and 1.6 JVM (on different OSes).

Lundy

Note that clueless commenters/naysayers may come up with non-sense like "you should never catch an OOM" but they've probably never worked on Real-World [TM] Java applications being deployed on hundreds of systems. I do this all the time and it just works. This has allowed me to remotely debug and fix lots of very subtile memory leak that never ever showed up during testing. Catching an OOM here makes perfect sense because the whole point is to understand why the OOM is happening. But don't be surprise to see lots of naysayers here don't understanding that very basic fact. – Lundy

Based on this answer, I created utility class UncaughtExceptionLogger (pastebin.com/e30g9y66). This one you can define, say, as a Spring bean, and you're all set with extended logging. – Revealment

Answer 2

If you're in a Linux/Unix environment you can do this:

-XX:OnOutOfMemoryError="kill -3 %p"

This way you don't have to have your application generate periodic thread dumps and you'll get a snapshot when it actually chokes.

With %p, you don't need to pass the PID, JVM will automatically pick the correct process id as mentioned here.

Lamanna

Actually, it turns out that you can just load your Heap Dump generated on OOME into VisualVM and click on the "Show Threads" link under the "Threads at the heap dump" section title. – Lamanna

-XX:OnOutOfMemoryError="kill -3 %p" - and you don't need to parse and specify the pid manually (forums.oracle.com/forums/…) – Lorilee

As to "-XX:OnOutOfMemoryError="kill -3 %p" can I take the thread dump to a specified log file? – Ganesha

Answer 3

It's possible to trigger a thread dump when OnOutOfMemoryError is triggered using jstack. e.g:-

jstack -F pid > /var/tmp/<identifier>.dump

Lucio

Answer 4

I don't think there is anything in java that would provide you with on-exit thread dumps. I tackle this when necessary by having a cronjob that does periodic kill -3 pid. Yes, it does clutter the logs a bit, but the footprint is still negligible.

And if you are suffering from OOM, it might be be beneficial to see how the situation evolved thread-wise.

Colossian

Answer 5

Based on the accepted answer I created utility class. This one you can define as a Spring bean and you're all set with extended logging.

import java.util.Iterator;
import java.util.Map;

import javax.annotation.PostConstruct;

import org.apache.commons.lang3.exception.ExceptionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class UncaughtExceptionLogger {

    private final static Logger logger = LoggerFactory.getLogger(UncaughtExceptionLogger.class);

    @PostConstruct
    private void init() {
        Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
            public void uncaughtException(final Thread t, final Throwable e) {
                String msg = ExceptionUtils.getRootCauseMessage(e);
                logger.error(String.format("Uncaght exception handler captured expcetion '%s'", msg), e);
                if (msg.contains("unable to create new native thread")) {
                    String dump = captureThreadDump();
                    logger.error(String.format(
                            "OutOfMemoryError has been captured for threads limit. Thread dump: \n %s", dump), e);
                }
                if (ExceptionUtils.getRootCause(e) instanceof OutOfMemoryError) {
                    String dump = captureThreadDump();
                    logger.error(String.format("OutOfMemoryError has been captured. Thread dump: \n %s", dump), e);
                }
            }
        });
    }

    public static String captureThreadDump() {
        /**
         * https://mcmap.net/q/758660/-how-to-generate-thread-dump-java-on-out-of-memory-error
         * dump-java-on-out-of-memory-error
         * http://henryranch.net/software/capturing-a-thread-dump-in-java/
         */
        Map<Thread, StackTraceElement[]> allThreads = Thread.getAllStackTraces();
        Iterator<Thread> iterator = allThreads.keySet().iterator();
        StringBuffer stringBuffer = new StringBuffer();
        while (iterator.hasNext()) {
            Thread key = (Thread) iterator.next();
            StackTraceElement[] trace = (StackTraceElement[]) allThreads.get(key);
            stringBuffer.append(key + "\r\n");
            for (int i = 0; i < trace.length; i++) {
                stringBuffer.append(" " + trace[i] + "\r\n");
            }
            stringBuffer.append("");
        }
        return stringBuffer.toString();
    }
}

Revealment

Answer 6

-XX:OnOutOfMemoryError="kill -3 %p"

The JVM argument to take thread dump sadly don't work. Child process cannot end SIGQUIT to parent.

Oracle has -XX:CrashOnOutOfMemoryError but this is available on Java 8.

Soredium

Answer 7

Assuming that you have a JDK (not a JRE) installed on your target server, you can rely on the command jcmd to generate your thread dump, this way it will work whatever the OS of your server.

The JVM option defining the command to execute on OutOfMemoryError is then:

-XX:OnOutOfMemoryError="jcmd %p Thread.print"

Where %p designates the current process id

Moreover, as we could want to have the thread dump into a file, it is possible to have it into the JVM log file by adding the JVM options -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=jvm.log, this way the heap dump will be available in the file jvm.log located in the working directory of the JVM along with other information intended for diagnosing the JVM.

The full list of JVM options to add is then:

-XX:OnOutOfMemoryError="jcmd %p Thread.print" -XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput -XX:LogFile=jvm.log

Be aware that the file is always created even when no OOME happens, so on a server, if you want to avoid having the previous JVM file being replaced at the next startup, you should consider adding the process id in the name of the log file like for example jvm_pid%p.log but don't forget to remove the files regularly.

If the JVM option -XX:LogFile is not set, by default the name of the file is of type hotspot_pid%p.log where %p designates the current process id.

---

In practice, the only JVM option that is needed is -XX:+HeapDumpOnOutOfMemoryError as it allows to generate an hprof file when an OnOutOfMemoryError occurs which already contains a thread dump.

See below how to get a thread dump from an hprof file using VisualVM:

Sharyl