"subprocess.Popen" - checking for success and errors

Question

I want to check if a subprocess has finished execution successfully or failed. Currently I have come up with a solution but I am not sure if it is correct and reliable. Is it guaranteed that every process outputs its errors only to stderr respectfully to stdout:

Note: I am not interested in just redirecting/printing out the output. That I know already how to do.

pipe = subprocess.Popen(command,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE,
                                universal_newlines=True)

if "" == pipe.stdout.readline():
    print("Success")
    self.isCommandExectutionSuccessful = True

if not "" == pipe.stderr.readline():
    print("Error")
    self.isCommandExectutionSuccessful = True

alternatively:

   if "" == pipe.stdout.readline():
       print("Success")
       self.isCommandExectutionSuccessful = True
   else:
       print("Error")
       self.isCommandExectutionSuccessful = False

and:

   if not "" == pipe.stderr.readline():
       print("Success")
       self.isCommandExectutionSuccessful = True
   else:
       print("Error")
       self.isCommandExectutionSuccessful = False

Protease

Answer 1

Do you need to do anything with the output of the process?

The check_call method might be useful here. See the python docs here: https://docs.python.org/2/library/subprocess.html#subprocess.check_call

You can then use this as follows:

try:
  subprocess.check_call(command)
except subprocess.CalledProcessError:
  # There was an error - command exited with non-zero code

However, this relies on command returning an exit code of 0 for succesful completion and a non-zero value for an error.

If you need to capture the output as well, then the check_output method may be more appropriate. It is still possible to redirect the standard error if you need this as well.

try:
  proc = subprocess.check_output(command, stderr=subprocess.STDOUT)
  # do something with output
except subprocess.CalledProcessError:
  # There was an error - command exited with non-zero code

See the docs here: https://docs.python.org/2/library/subprocess.html#subprocess.check_output

Juxtapose

Well I actually output stdout and stderr it to a window and then I need to check if there is an error and set a boolean variable that is used in another part of the program. I tried returncode but it is None regardless of success or error. – Protease

OK. There's also the check_output method. You can get the stdout and stderr from those and still catch the error. See here: docs.python.org/2/library/… – Juxtapose

Answer 2

Complete solution with check on return code, stdout and stderr:

import subprocess as sp

# ok
pipe = sp.Popen( 'ls /bin', shell=True, stdout=sp.PIPE, stderr=sp.PIPE )
# res = tuple (stdout, stderr)
res = pipe.communicate()
print("retcode =", pipe.returncode)
print("res =", res)
print("stderr =", res[1])
for line in res[0].decode(encoding='utf-8').split('\n'):
  print(line)

# with error
pipe = sp.Popen( 'ls /bing', shell=True, stdout=sp.PIPE, stderr=sp.PIPE )
res = pipe.communicate()
print("retcode =", pipe.returncode)
print("res =", res)
print("stderr =", res[1])

Prints:

retcode = 0
res = (b'bash\nbunzip2\nbusybox\nbzcat\n...zmore\nznew\n', b'')
stderr = b''
bash
bunzip2
busybox
bzcat
...
zmore
znew

retcode = 2
res = (b'', b"ls: cannot access '/bing': No such file or directory\n")
stderr = b"ls: cannot access '/bing': No such file or directory\n"

Torrietorrin

what are the possible values for returncode here? is it possible to set that code...im trying to figure out if anyway to know if another process i have that kill a running process, might be able to either set the return code or stderr before killing..so i can check and gracefully clean up. – Mcroberts

@Mcroberts the return code is set up by the process through its exit() call, normally 0 if there are no errors and the errno in case of errors, if it follows the Unix conventions. If you control the source you can customize it through exit(), else you want to check the man entry of the command in the section Exit status. – Torrietorrin

Answer 3

There are situations where using check_call is not a possibility. For example when you need to communicate with the process (e.g., passing input with communicate).

In this case, a simple solution is to mimic check_call manually. We can look at the Python source code to see what check_call is doing here and you can see it's simply checking the return code is 0 and if not it raises a CalledProcessError. It couldn't be simpler.

You may notice that check_call is not including stdout or stderr in the CalledProcessError even though it can accept them. This is because the process may not have captured them unless it was given subprocess.PIPE for stdout and stderr Popen parameters.

video_url = "http://.../sample/BigBuckBunny.mp4"

p = subprocess.Popen(
    [
        "ffplay",
        "-i", "-"
    ],
    stdin=subprocess.PIPE
)

p.communicate(video_url.encode())
if p.returncode != 0:
    raise subprocess.CalledProcessError(p.returncode, p.args)

The above is an example scenario where we need to PIPE input data (in this case an URL) to the process. It is not possible to write to stdin with check_call.

We simply mimic check_call with the last 2 lines.

Knout

Answer 4

      output,error=pipe.communicate()

This will wait for command to finish and give you output or error depending on the state of command.

Multiracial

Answer 5

You can check return code of the process using check_call() method. In case if process returned non-zero value CalledProcessError will be raised.

Axenic

@Zingam: p.returncode is None means that the process is still running. Note: p.returncode can't be None after p.communicate(). – Copra

Answer 6

This is how I did it finally:

    # Call a system process
    try:
        # universal_newlines - makes manual decoding of subprocess.stdout unnecessary
        output = subprocess.check_output(command,
                                         stderr=subprocess.STDOUT,
                                         universal_newlines=True)

        # Print out command's standard output (elegant)
        self.textEdit_CommandLineOutput.insertPlainText(output)
        self.isCommandExecutionSuccessful = True

    except subprocess.CalledProcessError as error:
        self.isCommandExecutionSuccessful = False

        errorMessage = ">>> Error while executing:\n"\
                       + command\
                       + "\n>>> Returned with error:\n"\
                       + str(error.output)
        self.textEdit_CommandLineOutput.append(errorMessage)

        QMessageBox.critical(None,
                             "ERROR",
                             errorMessage)
        print("Error: " + errorMessage)

    except FileNotFoundError as error:
        errorMessage = error.strerror
        QMessageBox.critical(None,
                             "ERROR",
                             errorMessage)
        print("Error: ", errorMessage)

I hope it will be useful to someone else.

Protease

it is incorrect. check_output() returns all output as a single string. for character in some_string yields characters in Python, not lines. Also you your GUI will be frozen while the child process is running. You should use threads or async. io to avoid blocking the GUI. See code examples in the links – Copra