Programmatic deadlock detection in java

Question

How can I programmatically detect that a deadlock has occurred in a Java program?

Refresh

You should specify whether you need to detect this programmatically or via some kind of external monitoring tool? – Neap

The JavaSpecialist website (always worth reading) has an interesting article on this, discussing the theory and practice. – Battista

Ah, my answer was assuming using a tool. I guess it is useful to know about both approaches anyway. – Tiffaneytiffani

Answer 1

You can do this programmatically using the ThreadMXBean that ships with the JDK:

ThreadMXBean bean = ManagementFactory.getThreadMXBean();
long[] threadIds = bean.findDeadlockedThreads(); // Returns null if no threads are deadlocked.

if (threadIds != null) {
    ThreadInfo[] infos = bean.getThreadInfo(threadIds);

    for (ThreadInfo info : infos) {
        StackTraceElement[] stack = info.getStackTrace();
        // Log or store stack trace information.
    }
}

Obviously you should try to isolate whichever thread is performing this deadlock check - Otherwise if that thread deadlocks it won't be able to run the check!

Incidentally this is what JConsole is using under the covers.

Chacha

The Javadoc for that method says it might be expensive. Do you happen to know about how expensive? – Kimmel

No I don't although I guess you could write some code to time the operation and see how this scales as thread count / number of locks increases. – Chacha

There are a few problems with this code. First, findDeadlockedThreads could return null for some threads which can cause NPE in your loop. Second, this will not print actual stack trace, you need to get actual threads from thread info. You can check this post for more info: korhner.github.io/java/multithreading/… – Padnag

@korhner: findDeadlockedThreads() will return null in the case of there being no deadlocked threads, which I already check for; I don't believe it ever returns an array containing sporadic null values. getThreadInfo can return an array containing nulls if the corresponding threads are not alive. However, I already know they are alive (and deadlocked) from the previous call. Re. the second point I've left stack trace logging as an exercise for the OP but they don't actually mention that they want to do this. – Chacha

@Adamski:findDeadlockedThreads can report deadlocks even if a thread has a timeout on a lock which in practice is not a real dedlock (after timeout lock is released). In theory it can happen that the thread is destroyed after timeout and before your getThreadInfo. You are right about findDeadlockedThreads won't return null for specific threads however, sorry. – Padnag

It is working for me in some cases. But not for all cases. – Aggappora

I benchmarked the findDeadlockedThreads method (on a HotSpot Server VM with only few threads, admittedly). Each call takes less than 100 µs after warmup. tinybrain.de/1009294 – Linctus

Benchmark results for more threads: For 100 sleeping threads the detection takes <250 µs. For 1000 sleeping threads it takes <4 ms. Still very OK to do occasionally. – Linctus

In case you wondered why stack trace is empty in above example: Use getThreadInfo(threadIds, Integer.MAX_VALUE) to acquire all stack trace elements. – Bartie

Answer 2

One useful hint for investigation:

If you can catch the application red handed and suspect a deadlock has occurred, go and press "Ctrl-Break" in the java.exe console window (or "Ctrl-\" on Solaris/Linux). The jvm will dump the current status and stack trace of all threads, find out dead locks and precisely describe them.

It will look something like this:

Full thread dump Java HotSpot(TM) Client VM (1.5.0_09-b03 mixed mode):

"[Test Timer] Request Queue" prio=6 tid=0x13d708d0 nid=0x1ec in Object.
    wait() [0x1b00f000..0x1b00fb68]
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Unknown Source)
    at library.util.AsyncQueue.run(AsyncQueue.java:138)
        - locked <0x02e70000> (a test.server.scheduler.SchedulerRequestQueue)

    ...

Found one Java-level deadlock:
=============================
"Corba service":
  waiting to lock monitor 0x13c06684 (object 0x04697d90, a java.lang.Object),
  which is held by "[Server Connection] Heartbeat Timer"
"[Server Connection] Heartbeat Timer":
  waiting to lock monitor 0x13c065c4 (object 0x0467e728, a test.proxy.ServerProxy), which is held by "Corba service"

Java stack information for the threads listed above:
===================================================
"Corba service":
    at test.proxy.ServerProxy.stopHBWatchDog(ServerProxy:695)
    - waiting to lock <0x04697d90> (a java.lang.Object)
    ...

Foliose

Or ctrl-\ from Solaris/Linux. – Lovett

Actually what you want is send a "QUIT" signal (ie a SIGQUIT) but it's not really a 'QUIT'. A SIGQUIT on OS X, Linux (and Solaris) on a Java app dumps the stacktrace. Ctrl+\ is one way to send a SIGQUIT. Taking the pid and doing: kill -3 {id} is another way to do it. – Bes

Answer 3

You can detect the deadlocked threads programmatically using ThreadMXBean class.Here is the code,

    ThreadMXBean bean = ManagementFactory.getThreadMXBean();

    long ids[] = bean.findMonitorDeadlockedThreads();

    if(ids != null)
    {
        ThreadInfo threadInfo[] = bean.getThreadInfo(ids);

        for (ThreadInfo threadInfo1 : threadInfo)
        {
            System.out.println(threadInfo1.getThreadId());    //Prints the ID of deadlocked thread

            System.out.println(threadInfo1.getThreadName());  //Prints the name of deadlocked thread

            System.out.println(threadInfo1.getLockName());    //Prints the string representation of an object for which thread has entered into deadlock.

            System.out.println(threadInfo1.getLockOwnerId());  //Prints the ID of thread which currently owns the object lock

            System.out.println(threadInfo1.getLockOwnerName());  //Prints name of the thread which currently owns the object lock.
        }
    }
    else
    {
        System.out.println("No Deadlocked Threads");
    }

Click here for more info on how to detect the deadlocked threads.

Wingless

Better to use findDeadlockedThreads() instead of the findMonitorDeadlockedThreads(). The findDeadlockedThreads has been introduced on 1.6 and is better in comparison with the findMonitorDeadlockedThreads() that may loose some deadlock cases and not report them. It is not able to report deadlocks for: ReentrankLock or ReentrantReadWriteLock. More info can be found here: meteatamel.wordpress.com/2012/03/21/deadlock-detection-in-java under the "findMonitorDeadlockedThreads vs. findDeadlockedThreads" title – Mischance

Answer 4

JArmus is a library for deadlock detection and avoidance. It includes support for: Thread.join, CyclicBarrier, CountDownLatch, Phaser, and ReentrantLock.

To use JArmus you need to instrument your code. Either through one of its instrumented classes or automatically with the JArmus instrumentar jarmusc.

java -jar jarmusc.jar yourprogram.jar checkedprogram.jar

The input yourprogram.jar is the program you want to check. The output is the same program with checks to automatically find any deadlock.

Barriers need some help

Verifying deadlocks with classes CyclicBarrier, CountDownLatch, Phaser is a bit tricky --- for example, JConsole cannot detect these types of deadlocks. JArmus needs a little help from you: you must specify which threads are influencing synchronization, we call these registered threads.

As soon as possible, the thread must mark itself as registered. A good place to mark registered threads is at the beginning method Runnable.run. JArmus.register(latch);

Example

The following program that deadlocks is correctly identified by JArmus:

final CountDownLatch latch = new CountDownLatch(2);
final CyclicBarrier barrier = new CyclicBarrier(2);
final Queue<Exception> exceptions = new ArrayDeque<>();
Thread t1 = new Thread(new Runnable() {
    @Override
    public void run() {
        try {
            JArmus.register(barrier); // do not forget to register!
            JArmus.register(latch); // do not forget to register!
            latch.countDown();
            latch.await();
            barrier.await();
        } catch (Exception e) {
            exceptions.add(e);
        }
    }
});
Thread t2 = new Thread(new Runnable() {
    @Override
    public void run() {
        try {
            JArmus.register(barrier); // do not forget to register!
            JArmus.register(latch); // do not forget to register!
            barrier.await();
            latch.countDown();
            latch.await();
        } catch (Exception e) {
            exceptions.add(e);
        }
    }
});
t1.start();
t2.start();

Ogawa

Answer 5

You might want to consider IBM's MTRAT. Prevention is better than cure after all. The Multicore Software Development Kit also comes with a deadlock detection tool.

Tiffaneytiffani

The "MTRAT" link goes to a page about floating-point arithmetic. Confused what that's about – Linctus

Answer 6

If you don't require programmatic detection you can do this via the JConsole; on the thread tab there is a "detect deadlock" button. In JDK6 this detect locks for both intrinsic monitors and j.u.c Locks

Run up the JConsole via the $JAVA_HOM/bin/jconsole command

Neap

Answer 7

tempus-fugit also implements it along with a programmatic thread dumping class. It's implemented using the mbean mechanism mentioned above and offers a drop in, out-of-the-box super duper solution.

Sanction

Answer 8

There is code here: http://www.java2s.com/Code/Java/Development-Class/PerformingdeadlockdetectionprogrammaticallywithintheapplicationusingthejavalangmanagementAPI.htm

The magic happens in ThreadMonitor.findDeadlock():

  public boolean findDeadlock() {
    long[] tids;
    if (findDeadlocksMethodName.equals("findDeadlockedThreads")
        && tmbean.isSynchronizerUsageSupported()) {
      tids = tmbean.findDeadlockedThreads();
      if (tids == null) {
        return false;
      }

      System.out.println("Deadlock found :-");
      ThreadInfo[] infos = tmbean.getThreadInfo(tids, true, true);
      for (ThreadInfo ti : infos) {
        printThreadInfo(ti);
        printLockInfo(ti.getLockedSynchronizers());
        System.out.println();
      }
    } else {
      tids = tmbean.findMonitorDeadlockedThreads();
      if (tids == null) {
        return false;
      }
      ThreadInfo[] infos = tmbean.getThreadInfo(tids, Integer.MAX_VALUE);
      for (ThreadInfo ti : infos) {
        // print thread information
        printThreadInfo(ti);
      }
    }

    return true;
  }

This calls an API of the ThreadMXBean which has a different name in Java 5 and 6 (hence the outer if()).

The code example also allows to interrupt the locks, so you can even break the deadlock.

Inflexed

Answer 9

In case you want it to be done in run-time you can use watchdog for that.

Onstad