Kill or terminate subprocess when timeout?

Question

I would like to repeatedly execute a subprocess as fast as possible. However, sometimes the process will take too long, so I want to kill it. I use signal.signal(...) like below:

ppid=pipeexe.pid
signal.signal(signal.SIGALRM, stop_handler)

signal.alarm(1)
.....
def stop_handler(signal, frame):
    print 'Stop test'+testdir+'for time out'
    if(pipeexe.poll()==None and hasattr(signal, "SIGKILL")):
         os.kill(ppid, signal.SIGKILL)
         return False

but sometime this code will try to stop the next round from executing. Stop test/home/lu/workspace/152/treefit/test2for time out /bin/sh: /home/lu/workspace/153/squib_driver: not found ---this is the next execution; the program wrongly stops it.

Does anyone know how to solve this? I want to stop in time not execute 1 second the time.sleep(n) often wait n seconds. I do not want that I want it can execute less than 1 second

Raffarty

So essentially if the subprocess runs over 1 second, you want to kill it and start the next one? Is this correct? – Please

How do you create your subprocess ? because it look like the expression ppid=pipeexe.pid is getting the next subprocess that will be run !!! – Emilemile

So essentially if the subprocess runs over 1 second, you want to kill it and start the next one? Is this correct? yes, that is right – Raffarty

How do you create your subprocess ? pipeexe=subprocess.Popen(...) – Raffarty

because it look like the expression ppid=pipeexe.pid is getting the next subprocess that will be run!! I think you may right. but how to make sure that the pid does not change when the next time run – Raffarty

actually, the code work all right at beginning but after it killed process several time. it is begin wrong – Raffarty

Answer 1

You could do something like this:

import subprocess as sub
import threading

class RunCmd(threading.Thread):
    def __init__(self, cmd, timeout):
        threading.Thread.__init__(self)
        self.cmd = cmd
        self.timeout = timeout

    def run(self):
        self.p = sub.Popen(self.cmd)
        self.p.wait()

    def Run(self):
        self.start()
        self.join(self.timeout)

        if self.is_alive():
            self.p.terminate()      #use self.p.kill() if process needs a kill -9
            self.join()

RunCmd(["./someProg", "arg1"], 60).Run()

The idea is that you create a thread that runs the command and to kill it if the timeout exceeds some suitable value, in this case 60 seconds.

Verniavernice

+1. This method is what I use in my build system for testing executables. – Retentivity

Also +1. This seems to be one of the cleanest methods I've seen so far. With minor modification works on Python < 2.6 on linux too using os.kill(self.p.pid, signal.SIGKILL) (or SIGTERM followed by SIGKILL). Can also replace self.p.wait() with self.out, self.err = self.p.communicate() to avoid blocking the subprocess if it fills up stdout/stderr pipes – Amarillo

I pasted this code in and my command appears to be starting properly - however, I need to run one thread at a time. Start a process, let it finish naturally or kill it if it takes too long, then start again. – Coquette

is it possible to get the return value of Popen from this? – Sweatband

@galois: After the Run() method has executed the self.start() statement, the RunCmd instance's self.p contains the subprocess.Popen instance that's created, so you could access it that way. Why do you want this? – Grochow

Answer 2

Here is something I wrote as a watchdog for subprocess execution. I use it now a lot, but I'm not so experienced so maybe there are some flaws in it:

import subprocess
import time

def subprocess_execute(command, time_out=60):
    """executing the command with a watchdog"""

    # launching the command
    c = subprocess.Popen(command)

    # now waiting for the command to complete
    t = 0
    while t < time_out and c.poll() is None:
        time.sleep(1)  # (comment 1)
        t += 1

    # there are two possibilities for the while to have stopped:
    if c.poll() is None:
        # in the case the process did not complete, we kill it
        c.terminate()
        # and fill the return code with some error value
        returncode = -1  # (comment 2)

    else:                 
        # in the case the process completed normally
        returncode = c.poll()

    return returncode   

Usage:

 return = subprocess_execute(['java', '-jar', 'some.jar'])

Comments:

1. here, the watchdog time out is in seconds; but it's easy to change to whatever needed by changing the time.sleep() value. The time_out will have to be documented accordingly;
2. according to what is needed, here it maybe more suitable to raise some exception.

Documentation: I struggled a bit with the documentation of subprocess module to understand that subprocess.Popen is not blocking; the process is executed in parallel (maybe I do not use the correct word here, but I think it's understandable).

But as what I wrote is linear in its execution, I really have to wait for the command to complete, with a time out to avoid bugs in the command to pause the nightly execution of the script.

Tevis

Answer 3

I guess this is a common synchronization problem in event-oriented programming with threads and processes.

If you should always have only one subprocess running, make sure the current subprocess is killed before running the next one. Otherwise the signal handler may get a reference to the last subprocess run and ignore the older.

Suppose subprocess A is running. Before the alarm signal is handled, subprocess B is launched. Just after that, your alarm signal handler attempts to kill a subprocess. As the current PID (or the current subprocess pipe object) was set to B's when launching the subprocess, B gets killed and A keeps running.

Is my guess correct?

To make your code easier to understand, I would include the part that creates a new subprocess just after the part that kills the current subprocess. That would make clear there is only one subprocess running at any time. The signal handler could do both the subprocess killing and launching, as if it was the iteration block that runs in a loop, in this case event-driven with the alarm signal every 1 second.

Gabo

Answer 4

Here's what I use:

class KillerThread(threading.Thread):
  def __init__(self, pid, timeout, event ):
    threading.Thread.__init__(self)
    self.pid = pid
    self.timeout = timeout
    self.event = event
    self.setDaemon(True)
  def run(self):
    self.event.wait(self.timeout)
    if not self.event.isSet() :
      try:
        os.kill( self.pid, signal.SIGKILL )
      except OSError, e:
        #This is raised if the process has already completed
        pass    

def runTimed(dt, dir, args, kwargs ):
  event = threading.Event()
  cwd = os.getcwd()
  os.chdir(dir)
  proc = subprocess.Popen(args, **kwargs )
  os.chdir(cwd)
  killer = KillerThread(proc.pid, dt, event)
  killer.start()

  (stdout, stderr) = proc.communicate()
  event.set()      

  return (stdout,stderr, proc.returncode)

Reportage

Answer 5

A bit more complex, I added an answer to solve a similar problem: Capturing stdout, feeding stdin, and being able to terminate after some time of inactivity and/or after some overall runtime.

Conjure

Answer 6

subprocess.run was added in Python 3.5, meaning most systems should already have access to an up-to-date Python version with this method. It makes it trivial to set the timeout.

Here's an example to run your-executable (assuming it's in the working directory) with --optional args arguments and a timeout of 60 seconds:

import subprocess

subprocess.run(['./your-executable', '--optional', 'args'], timeout=60)

run will raise subprocess.TimeoutExpired if the timeout expires, which you could also handle:

try:
    subprocess.run(['./your-executable', '--optional', 'args'], timeout=60)
except subprocess.TimeoutExpired:
    print('timed out!')

Dummy