subprocess.check_output with grep command fails when grep finds no matches

Question

I'm trying to search a text file and retrieve lines containing a specific set of words. This is the code I'm using:

tyrs = subprocess.check_output('grep "^A" %s | grep TYR' % pocket_location, shell = True).split('\n')

This works fine when the file contains at least one line that grep identifies. But when grep doesn't identify any lines, grep returns exit status 1 and I get the following error:

Traceback (most recent call last):
  File "../../Python_scripts/cbs_wrapper2.py", line 324, in <module>
    tyrs = subprocess.check_output('grep "^ATOM" %s | grep TYR' % pocket_location, shell = True).split('\n')
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'grep "^ATOM" cbsPrediction_files/1u9c_clean/1u9c_clean_fpocket_out/pockets/pocket0_atm.pdb | grep TYR' returned non-zero exit status 1

How can I avoid this issue? I just want subprocess.check_output to return an empty string if grep doesn't find anything.

Thanks

Tine

use pipes.quote(pocket_location) to allow the path with characters that are special in a shell such as space character. – Sherasherar

Answer 1

I just want subprocess.check_output to return an empty string if grep doesn't find anything.

Well, too bad. grep considers no matches to be failure, and the whole point of the check in check_output is to check for failure, so you're explicitly asking to do things this way. Here are the relevant docs:

If the return code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and any output in the output attribute.

And for grep:

The following exit values shall be returned:
  0 One or more lines were selected.
  1 No lines were selected.
  >1 An error occurred.

So, if you want to treat "no lines" as success, but actual errors as errors, you have to handle that 1 value differently than other non-zero values. And check_output has no idea that you want to do that.

So, either you have to handle the CalledProcessError, or you have to do your own checking. In other words, either this:

try:
    tyrs = subprocess.check_output('grep "^A" %s | grep TYR' % pocket_location, shell = True).split('\n')
except subprocess.CalledProcessError as e:
    if e.returncode > 1:
        raise
    tyrs = []

… or this:

p = subprocess.Popen('grep "^A" %s | grep TYR' % pocket_location, shell=True,
                     stdout=subprocess.PIPE)
output, _ = p.communicate()
if p.returncode == 1: # no matches found 
    tyrs = []
elif p.returncode == 0: # matches found
    tyrs = output.split('\n')
else:
    # error, do something with it

Tallyho

Answer 2

tyrs = subprocess.check_output('grep "^A" %s | grep TYR || true' % pocket_location, shell = True).split('\n')

Sigmon

The shell command is ended with call to true which ensures the return code is always 0 – Enthalpy

Of course, if the grep fails for any other reason you are screwed with this approach. I suggest to take the abarnert solution if you are serious about programming. – Enthalpy

Answer 3

For future reference, if anyone is looking for a similar solution using pgrep -

import subprocess    
var = subprocess.check_output("pgrep -af 'python3 choice_multi.py CMC0001'", shell=True, text=True)
var = var.splitlines()
var = [x for x in var if '/bin/sh' not in x]
print(var )

This will come as an irritating error if you do pgrep because it also shows the PID of the temporary process being created.

Now the /bin/sh can be different for other systems. Alternatively, there are two other solutions that I thought as inefficient -

1. You can just delete the first item in the list.
2. You can check the length of the list and scan for the above 1 only because all of them will have a minimum 1 PID.

References - Is there a simple way to delete a list element by value?

Hypognathous