Thread-launched running processes won't destroy (Java)

Question

Starting multiple threads and having each exec() then destroy() a running java process result in some of the process not being destroyed and still running after program exit. Here is some code that reproduce the issue. I noticed the more threads you start, the more processes stay alive. And the more sleep before destroy(), the less processes stay alive. (I used InfiniteLoop as an example. Any running process will do the trick.)

EDIT : Bug has been reported to Oracle, waiting for an answer. Feel free to share any knowledge/experiments on the subject.

for(int i = 0; i < 100; i++)
{
  new Thread(new Runnable()
  {
    public void run()
    {
      try
      {
        Process p = Runtime.getRuntime().exec(new String[]{"java", "InfiniteLoop"});
        Thread.sleep(1);
        p.destroy();
      }catch(IOException | InterruptedException e){e.printStackTrace();}                    
    }
  }).start();
}

Wanderlust

Are you getting exceptions? – Abramson

There is no exception. – Wanderlust

e.printStackTrace() in the catch block does nothing during execution. – Wanderlust

Are you waiting for the threads to terminate before exiting? Are you spawning them from a daemon thread (if so then perhaps the JVM is terminating them inside their sleep() call when your program exits)? – Lavona

I'm going to try and recreate this. Am I correct in assuming that InfiniteLoop just runs an infinite loop? – Lavona

I just wait for the program to end (when every thread has terminated i guess.) It doesn't block. And I start them from the main thread. InfiniteLoop is just an infinite loop. – Wanderlust

I am unable to reproduce the behavior you are describing. When I run this, 100 java subprocesses start, then they end. I'm testing this on Ubuntu. What platform are you on? Can you post your entire test program? – Lavona

@JasonC Is there anymore java processes with ps -e ? I use Mint. Test program is just the above code inside a main method. InfiniteLoop class is just while(true){} inside a main method. – Wanderlust

@basbodart No, there are no processes remaining. – Lavona

I posted the source I am using below, and a few more observations. It's not intended to be an answer, just a temporary place for discussion. – Lavona

I just had it leave about 5 processes running after it terminated. It doesn't always happen. Weird. I want to know more about this too. I have a hunch that it has something to do with destroying the process too quickly or some kind of race condition; maybe java forks something off or does something to create a new process that destroy() doesn't take care of if called too quickly / at the wrong time. – Lavona

I found an old bug (but it is not mark resolved) stating that if a process spawns subprocesses they may not be killed by destroy(). bugs.sun.com/bugdatabase/view_bug.do?bug_id=4770092 What version of the JDK are you using. – Lavona

Here's another reference to what looks like a similar issue: #4912782 And I want to apologize if I've only added confusion to your life, I don't actually use Process that much and am not familiar with the quirks. Hopefully somebody else will step in with a definitive answer. It seems like it doesn't handle subprocesses well, and I'm presuming java forks something off. That's all I got. – Lavona

What is this for? If your test is representative of a real application you are running, perhaps you have other alternatives for communicating with child processes and telling them to quit. – Lavona

I think it's about destroying too quickly too. But why would java needs to fork a brand new process? That's strange. I updated java to the last JDK7u25 from Oracle to no avail. I'm doing automated code testing which implies I might be dealing with infinite looping methods. Since I can't modify the code to handle Thread.interrupt() and Thread.stop() is deprecated (and might not even work in every case), Process.destroy() is the only option I have. – Wanderlust

I submitted a bug report to Oracle, waiting for an answer. – Wanderlust

Post back here if they reply, very curious. – Lavona

@basbodart, is behavior same on different Operating Systems. Also is there any change in behavior if the system resources like CPU/RAM are increased? IMHO this may not be a Java problem per se. – Amendment

How are you determining that your program has exited before the child processes terminate? – Oidium

@Amendment I have no idea, I didn't test. Just had one shot at it on a virtual box windows 7 64 and I saw no problem then but I did not explore it in depth. – Wanderlust

@Oidium Because there are still java processes in the table but the main one isn't – Wanderlust

Try to catch Throwable. May be you are missing something. – Wallen

@Noofiz No throwable is catched. – Wanderlust

This is OS dependent behavior. I dont think this is a bug. destroy() just send a signal to process, but for some reason OS does not fulfill it. May be this can help #10630803 – Wallen

@Noofiz Even if it is OS dependent, this behavior violates specification of the destroy() method which is the definition of a bug. JVM have to comply with the OS, not the opposite. As for the link, I already read that. Indeed a SIGTERM is supposed to be sent to the process. Whether it isn't sent or acknowledged properly is the underlying reason of the problem I guess. – Wanderlust

I would suggest this be moved to the Java bug/ticket as it would be of more use there. Interesting bug though, if I had to take a stab I would say that this would be tied to the specific JVM/OS implementation. – Hammerhead

Did Oracle ever respond to that bug report? – Lavona

I received a response saying it was a new bug with ID 9005842 but it never appeared in the bug DB. – Wanderlust

@BastienBodart I too have this issue, I have threads creating and monitoring processes, and they instruct those processes to destroy() or for JAVA 1.8 destroyForcibly() to no avail whatsoever after 60 seconds of heavy duty processing. So instead of having 12 processes running I end up with more and more as time goes on. – Hereford

@Hereford I haven't looked at this issue for a while but after a few test it seems to be still present (Java 8u91, Mint 17.1). Except that when I use destroyForcibly() instead of destroy(), all processes are terminated, as per specification since I create them with a Runtime.exec() call. How are your processes created? Which system are you using? – Wanderlust

@BastienBodart It's OpenSuse 13.2 and 12.1, with 1.7(destrory()) and 1.8(either/or) to no avail. It's threads kicking off processes which kick off more processes. I do know that there is an additional bug in Java with processes kicking off processes. Not my code. We're actually going to run out code on another distro and see what happens since the actual behavior of destroy() is system implemented. – Hereford

Answer 1

Use a p.waitFor(); before p.destroy(); ,

this will ensure the completion of the previous process. I think you p.destroy command gets invoked sooner than the exec() command performs the action. Therefore it becomes useless.

Theology

p.waitFor() waits for the process to terminate, so in the case of the InfiniteLoop, it would never return. The exec() point -- I wonder; I can find nothing in the documentation that guarantees that the process has actually been started when exec() returns, although I assume it has been because it has the ability to e.g. throw errors when the command isn't found. – Lavona

doesnt it wait for the current process to return? and then p.destroy kills the process..it completes one creation and destruction cycle in one iteration.. – Theology

p.waitFor() waits for exec() to return, therefore using this as a marker..you may check whethr exec() works or not.. – Theology

See docs.oracle.com/javase/7/docs/api/java/lang/… – Lavona

Answer 2

If subprocesses write anything to stdout or stderr (intentionally or not), that could cause trouble:

"Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock."

Source: http://www.javaworld.com/jw-12-2000/jw-1229-traps.html

The whole article is IMO worth reading if you need to use Runtime.exec().

Iatric

It doesn't write anything, it's just an empty while(true){} – Wanderlust

Answer 3

You should close the input/output/error streams to the process. We saw some issues in the past where the forked process was not completing properly due to those streams not being closed (even if they weren't being used).

An exemplary solution:

p.destroy();
p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();

Sharmainesharman

This shouldn't be necessary, if you look e.g. at the implementation of UnixProcess#destroy(), it closes all the streams automatically. – Robin

@Robin - this question was answered 2 years ago. are you looking at a production version of java from two years ago? (and this experience may be from early jdk 6 or even 5, tough to keep them all straight at this point in time). – Sharmainesharman

Right, I was looking at JDK 7 to be precise. – Robin

The issue with java processes not throw away handle on Windows when all 3 streams are not closed manually persist even in Java 14. – Bucolic

Answer 4

This is simply because before the threads execute the destroy call, your main program terminates and all the associated threads leaving the started processes running. To verify this, simply add a System.out call after the destroy and you will find it is not executed. To overcome this add a Thread.sleep at the end of your main method and you will not have the orphaned processes. The below does not leave any process running.

public class ProcessTest {

public static final void main (String[] args) throws Exception {

    for(int i = 0; i < 100; i++) {
        new Thread(new Runnable()
            {
                public void run() {
                    try {
                        Process p = Runtime.getRuntime().exec(new String[]{"java", "InfiniteLoop"});
                        Thread.sleep(1);
                        p.destroy();
                        System.out.println("Destroyed");
                    }catch(IOException e) {
                        System.err.println("exception: " + e.getMessage());
                    } catch(InterruptedException e){
                        System.err.println("exception: " + e.getMessage());
                    }
                }
            }).start();
    }


    Thread.sleep(1000);

}

}

Verda

That doesn't resolve the issue; that's actually one of the first tests I ran too. I had a 10 second sleep, and during that time, a handful of the processes remained running during the entire sleep even though they should've been destroyed (and didn't terminate when the program terminated either). – Lavona

I am running the above code in Debian and it has no processes left after adding Thread.sleep(). What I have done above is not the correct way to wait for threads to stop. You will have to call Thread.join on each of the threads created in a loop at the end. Have you tried this and is it still the same problem? – Verda

Tried your code, problem remains. But I don't see your point, the problem does not come from the threads but from the processes. A terminating thread does not terminate the processes it launched. – Wanderlust

Agreed, the started processes are not terminating. But there are two reasons for not terminating, it is not terminating because destroy is not called or destroy itself has a bug. To eliminate the first, I suggested that you join the main thread with the started threads, so you are sure that the threads have terminated which eliminates that scenario. Calling "sleep" method does not eliminate any scenario. – Verda

Further researching shows the following results: on Debian "jre build 1.7.0-b147" / "jdk build 1.6.0_31-b04" and on windows 7 32-bit "java build 1.6.0_31-b04", the processes are getting terminated correctly. But on a windows 7 64-bit using a jdk build 1.7.0-b147, the processes are not getting terminated even if destroy is called. – Verda

Answer 5

I believe that according to link, a distinct process is spawned by the operating system in response to this call. This process has a lifetime independent of your Java program and threads within it so you would expect it to continue running after your program has exited. I just tried it on my machine and it appeared to work as expected:

import java.io.*;

class Mp {
public static void main(String []args) {
    for(int i = 0; i < 100; i++) {
        new Thread(new Runnable() {
            public void run() {
                try {
                    System.out.println("1");
                    Process p = Runtime.getRuntime().exec
                        (new String[]{"notepad", ""});
                    System.out.println("2");
                    Thread.sleep(5);
                    System.out.println("3");
                    p.destroy();
                    System.out.println("4");
                }
                catch(IOException | InterruptedException e) {
                    e.printStackTrace();
                }                    
            }
        }).start();
    }
}
}

Lunsford

Good point - I forgot about the destroy call. It would be useful to add a e.printStackTrace() to the exception block to see if any errors are occurring. – Lunsford

It would also be useful to add some println traces to the threads to see if destroy() is actually getting reached. – Lavona

I tried it out on my machine just now as follows with no luck - I wonder if the JVM instance on the OP's machine has an issue. – Lunsford

@AliB Could you try with an infinite-loop process? I only experience the issue with actually running processes. – Wanderlust

Answer 6

This solution worked for me:

Process p = Runtime.getRuntime().exec(new String[]{"java", "InfiniteLoop"});
p.descendants().forEach(ProcessHandle::destroy);
p.destroy();

You have to also destroy children processes.

Scheldt

Answer 7

There is a race condition between the time Runtime.exec kicks off a new thread to start a Process and when you tell that process to destroy itself.

I'm on a Linux machine, so I will use the UNIXProcess.class file to illustrate.

Runtime.exec(...) will create a new ProcessBuilder and start it which on a Unix machine creates a new UNIXProcess instance. In the constructor of UNIXProcess there is this block of code which actually executes the process in a background (forked) thread:

java.security.AccessController.doPrivileged(
            new java.security.PrivilegedAction() {
    public Object run() {
    Thread t = new Thread("process reaper") {
        public void run() {
                    try {
                        pid = forkAndExec(prog,
                      argBlock, argc,
                      envBlock, envc,
                      dir,
                      redirectErrorStream,
                      stdin_fd, stdout_fd, stderr_fd);
                    } catch (IOException e) {
                        gate.setException(e); /*remember to rethrow later*/
                        gate.exit();
                        return;
                    }
                    java.security.AccessController.doPrivileged(
                    new java.security.PrivilegedAction() {
                        public Object run() {
                        stdin_stream = new BufferedOutputStream(new
                                                FileOutputStream(stdin_fd));
                        stdout_stream = new BufferedInputStream(new
                                                FileInputStream(stdout_fd));
                        stderr_stream = new FileInputStream(stderr_fd);
                        return null;
                    }
                    });
                    gate.exit(); /* exit from constructor */
        int res = waitForProcessExit(pid);
        synchronized (UNIXProcess.this) {
            hasExited = true;
            exitcode = res;
            UNIXProcess.this.notifyAll();
        }
        }
    };
            t.setDaemon(true);
            t.start();
    return null;
    }
});

Notice that the background thread sets the field pid which is the Unix process id. This will be used by destroy() to tell the OS which process to kill.

Because there isn't any way to make sure that this background thread has run when destroy() is called, we may try to kill the process before it has run OR we may try to kill the process before pid field has been set; pid is uninitialized and therefore is 0. So I think calling destroy too early will do the equivalent of a kill -9 0

There is even a comment in the UNIXProcess destroy() that alludes to this but only considers calling destroy after the process has already finished, not before it has started:

// There is a risk that pid will be recycled, causing us to
// kill the wrong process!  So we only terminate processes
// that appear to still be running.  Even with this check,
// there is an unavoidable race condition here, but the window
// is very small, and OSes try hard to not recycle pids too
// soon, so this is quite safe.

The pid field is not even marked as volatile so we may not even see the most recent value all the time.

Greengrocer

This race condition doesn't exist here. The UNIXProcess constructor doesn't return until after the forkAndExec call in the thread completes -- that's what gate is for. In fact, that's how it is able to throw an exception if the process fails to start, you'll note that exception originates from that background thread. So the constructor will not return until the process is started, and therefore it is not possible to call destroy() before the process is started. – Lavona

The comment you see in destroy() is referring to a different scenario. That comment refers to the fact that if a process dies on its own, the OS may reuse its PID for some other new process, so by the time our application calls destroy(), the PID may be a different unrelated process that we don't want to kill. The code near that comment just says "if the process already exited on its own, don't kill it", which is an attempt (not 100% reliable, as noted) to avoid killing the wrong thing. – Lavona

Answer 8

I had a very similar issue and the problem with destroy() not working was manifesting even with a single thread.

Process process = processBuilder(ForeverRunningMain.class).start()
long endTime = System.currentTimeMillis() + TIMEOUT_MS; 
while (System.currentTimeMillis() < endTime) {
    sleep(50);
}
process.destroy();

The process was not always destroyed if TIMEOUT_MS was too low. Adding an additional sleep() before destroy() fixed it (even though I don't have an explanation why):

Thread.sleep(300);
process.destroy();

Robin