How to abort long running invokeMethod on com.sun.jdi.ObjectReference?

Question

I have my own JDI Debugger which calls the toString method on some objects:

com.sun.jdi.ObjectReferenceobject object = ...
ThreadReference threadRef = frameProxy.threadProxy().getThreadReference();
Value value = object.invokeMethod(threadRef, toStringMethod,
                    Collections.EMPTY_LIST, ObjectReference.INVOKE_SINGLE_THREADED);

The problem is that even if no breakpoints are set inside the toString() method, the invokeMethod never terminates so my debugger hangs. For example this happens when I call this on a Double object.

How can I kill the execution of invokeMethod after some duration of time?

Update: I tried implementing my own Double object and put some System.out.println() statements at the start and end of the toString() method and it seemed that the method is executed just fine but for some reason the debugger doesn't receive the result. Maybe this is a bug in JDI because there are many such bugs, but I am not looking for a solution for this, I am just looking for a way to abort the execution of invokeMethod() if it takes too much time.

Update2: I tried what ThierryB was suggesting but I can only invoke frameProxy.threadProxy().stop(object); in the manager thread. And the manager thread is blocked because of invokeMethod() so it won't execute my command. I tried something like this:

boolean[] isFinished = new boolean[2];
isFinished[0] = false;

DebuggerManagerThreadImpl managerThread = debugProcess.getManagerThread();
new Thread(() - > {
    try {
        Thread.sleep(2000);
        if (!isFinished[0]) {
            System.out.println("Invoked");
            managerThread.invokeCommand(new DebuggerCommand() {
                @Override
                public void action() {
                    try {
                        frameProxy.threadProxy().stop(object);
                    } catch (InvalidTypeException e) {
                        e.printStackTrace();
                    }
                    int threadStatus = frameProxy.threadProxy().status();
                    switch (threadStatus) {
                        case ThreadReference.THREAD_STATUS_RUNNING:
                            System.out.println("The thread is running.");
                            break;
                        case ThreadReference.THREAD_STATUS_ZOMBIE:
                            System.out.println("The thread has been completed.");
                            break;
                        case ThreadReference.THREAD_STATUS_WAIT:
                            System.out.println("The thread is waiting.");
                            break;
                        default:
                            System.out.println("The thread is not running / not waiting / not completed : but what is it doing right now ? (a little programmer joke ;) )");
                    }
                }

                @Override
                public void commandCancelled() {

                }
            });
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}).start();

Value value = object.invokeMethod(threadRef, toStringMethod,
    Collections.EMPTY_LIST, ObjectReference.INVOKE_SINGLE_THREADED);

isFinished[0] = true;

but frameProxy.threadProxy().stop(object); is never executed because the DebuggerCommand's action method is not called(thread is blocked).

Also this is a Stack trace when my Debugger hangs and I forcefully stop the process:

com.sun.jdi.VMDisconnectedException
    at com.jetbrains.jdi.TargetVM.waitForReply(TargetVM.java:317)
    at com.jetbrains.jdi.VirtualMachineImpl.waitForTargetReply(VirtualMachineImpl.java:1170)
    at com.jetbrains.jdi.PacketStream.waitForReply(PacketStream.java:86)
    at com.jetbrains.jdi.JDWP$ObjectReference$InvokeMethod.waitForReply(JDWP.java:4840)
    at com.jetbrains.jdi.ObjectReferenceImpl.invokeMethod(ObjectReferenceImpl.java:413)

UPDATE 3: which thread to use to invoke the Method? Currently I am using frameProxy.threadProxy().getThreadReference(); which works fine most of the time, but is it better to create a separate thread just for invoking methods on objects(along my JDI debugger I also have an instrumentation agent inside the application so I can create a separate thread just for this use case (maybe this will prevent deadlocks?)).

UPDATE 4: Currently I am using SUSPEND_ALL as suspend policy, would it be better to use SUSPEND_EVENT_THREAD instead?

Marybellemarybeth

Be sure to carefully read the description paragraph of the javadoc on the invokeMethod [ docs.oracle.com/javase/8/docs/jdk/api/jpda/jdi/com/sun/jdi/… ] it details many conditions on which the target vm may encounter a deadlock. – Gagne

Yeah, I've ran into this type of hang too. It can be very frustrating. In my case I get no VMDisconnectedException whatsoever, which made it even harder to figure out what the cause was. – Forehead

@Gagne The truly relevant bit this "By default, when the invocation completes, all threads in the target VM are suspended, regardless their state before the invocation." This can cause hangs if you rely (only) on something like eventSet.resume() because the suspends caused by invokeMethod are not recorded in the eventSet you're processing, they're 'extra'. And the way the JVM works, you can suspend a thread multiple times stacked on each other. It uses a counter for suspends and counts/subtracts the number of resumes, which need to be >= num of suspends for resume to really work! – Forehead

@simonarame: The javadoc even delves into that saying "It can also cause a deadlock if invokeMethod is called from the client's event handler thread. [...] To avoid this, all EventRequests should be disabled before doing the invokeMethod, or the invokeMethod should not be done from the client's event handler thread." Which is alas what I was doing... – Forehead

In the OP's case though, the hang is because they're doing INVOKE_SINGLE_THREADED, which is documented to hang as well. "The resumption of other threads during the invocation can be prevented by specifying the INVOKE_SINGLE_THREADED bit flag in the options argument; however, there is no protection against or recovery from the deadlocks described above, so this option should be used with great caution." – Forehead

Answer 1

You can use the interface ThreadReference with the method void stop(ObjectReference throwable). The javadoc api tells "Stops this thread with an asynchronous exception.".

try {
    com.sun.jdi.ObjectReferenceobject object = ...
    ThreadReference threadRef = frameProxy.threadProxy().getThreadReference();
    frameProxy.threadProxy().stop(object);
    int threadStatus = frameProxy.threadProxy().status();
    switch(threadStatus) {
         case ThreadReference.THREAD_STATUS_RUNNING :
              log.info("The thread is running.");
              break;
         case ThreadReference.THREAD_STATUS_ZOMBIE :
              log.info("The thread has been completed.");
              break;
         case ThreadReference.THREAD_STATUS_WAIT :
              log.info("The thread is waiting.");
              break;
         default :
              log.info("The thread is not running / not waiting / not completed : but what is it doing right now ? (a little programmer joke ;) )");
    }
} catch (ClassNotLoadedException cnle) {
    // log exception, or display a message...
} catch (IncompatibleThreadStateException itse) {
    // log exception, or display a message...
} catch (InvalidTypeException ite) {
    // log exception, or display a message...
}

When you are checking your thread status with the status method, you can find that values :

• THREAD_STATUS_UNKNOWN
• THREAD_STATUS_ZOMBIE
• THREAD_STATUS_RUNNING
• THREAD_STATUS_SLEEPING
• THREAD_STATUS_MONITOR
• THREAD_STATUS_WAIT
• THREAD_STATUS_NOT_STARTED

Hope you will find a way to do what you need to with the stuff I have provided here. Examples from here or the following suite test published on the website of Alvin Alexander could be helpfull too.

Hurwitz

Thank you. Can you please take a look at my second update. I am sure that I am missing something. – Marybellemarybeth

@Nfff3: Well, you shouldn't be doing INVOKE_SINGLE_THREADED. That's documented to [guarantee] hang if the invocation needs to acquire any monitors held by other (suspended) threads in the debugee VM. Something as 'trivial' as invoking toString in the debugee can hang on some types of objects (in that fashion) because some (java.*) classes are coded as to not 'print' inconsistent internal states via toString, so they acquire monitor locks beforehand. In my experience that happens at least with Swing/AWT objects, but they're probably many other kinds that do that. – Forehead

@Nfff3: it is safe to decapsulate Booleans and similarly simple objects that way though. Those won't cause a hang (when their toString is invoked like that), again, in my experience. But something as 'complicated' as a StringBuffer will hang sometimes though because of monitor lock in its toString. – Forehead

Answer 2

object.invokeMethod( ... , toStringMethod, ... INVOKE_SINGLE_THREADED)

This is a bit of on X/Y problem.

The hangs you're seeing is because you're doing INVOKE_SINGLE_THREADED, which is documented to hang in some circumstances, as the javadoc says:

The resumption of other threads during the invocation can be prevented by specifying the INVOKE_SINGLE_THREADED bit flag in the options argument; however, there is no protection against or recovery from the deadlocks described above, so this option should be used with great caution.

Something as 'trivial' as invoking toString in the debugee can hang (in these circumstances) on some types of objects because some (java.*) classes are coded as to not 'print' inconsistent internal states via toString, so they acquire monitor locks beforehand. In my experience that happens at least with Swing & AWT objects, but they're probably many other kinds that do that.

It is safe to decapsulate java.lang.Booleans and similarly simple objects that way though. Those won't cause a hang when their toString is invoked like that, again, in my experience. But something as 'complicated' as a StringBuffer will hang sometimes though because of monitor lock in its toString.

If you only need to toString some simple objects ('simple' as defined above), then one way is to check their (mirrored) types (in the debugger). Personally, I check against a whitelist of strings for classes that don't cause hangs, e.g. with code like

String vstr = value.toString();
if (vstr.contains("instance of java.lang.Boolean")) {
  // safe to call toString via invokeMethod
}

but there are probably more elegant solutions.

Doing deferred invokeMethod ... toString printing from another thread may or may not capture the state of those objects when/where you care about them because they can change in the meantime. Making your event-handling thread disable all the pending requests while doing invokeMethod, which is another suggested workaround in the javadoc, has the downsides that you can miss some events. So, I think there's no universal workaround for this. Aborting/killing debuggee threads (which is what the bountied answer shows you how to do) is probably last on my list of workarounds though. (First, you'd need to make sure you're not killing an otherwise useful thread in the debugee, changing its behavior beyond merely aborting the invokeMethod you're doing. Second, debugger-owned [i.e. libjdwp's] threads in the debugee cannot actually be killed that way, i.e. with ThreadReference.stop because libjdwp ignores the stop requests for those threads.)

Finally, also on the XY angle, rather than invoking toString in the debugee (if that's the only kind of info you need), inspecting the Fields of the object(s) in question is a more robust solution, albeit the printing format will not be 100% as the debugee would do it with toString. I have tested his, and the running speed is roughly similar when fetching all the fields of objects compared to invoking toString on them in the debugee. The code you have to write is slightly more laborious -- about 50 lines of code for printing/decoding (as you have to tell apart arrays, and also look for a value field if you want to simplify the representation of most boxed objects), but it avoids all the trapdoors that invokeMethod has, deadlock-wise. It's also a far more tested/robust codepath JDK-internals-wise, as most visual/IDE debuggers fetch fields. So you're less likely to run into any unresolved JDK/JDI bugs, as well.

Forehead

Also of some relevance to the event-handler issued invokeMethod code.googlesource.com/edge/openjdk/+/jdk8u111-b14/jdk/test/com/… – Forehead

Apparently though there are some IDE debuggers that call toString or even hashCode on the debugee objects... and that can lead to some baffling errors for some users due to the brittleness of this approach https://mcmap.net/q/258390/-com-sun-jdi-invocationexception-occurred-invoking-method – Forehead