Subprocess command shows FileNotFoundError: [Errno 2] No such file or directory

Question

I'm trying to run shell commands using python by using subprocess module in the below code, but I don't why my script is throwing an error like below. Can someone help me what I'm missing?

Traceback (most recent call last):
  File "/Scripts/test_file.py", line 6, in <module>
    p2 = subprocess.Popen('sed s\'/&quot;/ /g\'', stdin=p1.stdout, stdout=subprocess.PIPE)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: "sed s'/&quot;/ /g'"`

---

import subprocess
#output3.txt='/Users/boggulv/Desktop/output3.txt'
p1 = subprocess.Popen( ['cat', 'output3.txt'], stdout=subprocess.PIPE)
print(p1)
p2 = subprocess.Popen('sed s\'/"/ /g\'', stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen('grep "sO"', stdin=p2.stdout,stdout=subprocess.PIPE)
p4 = subprocess.Popen('grep -v "get"', stdin=p3.stdout, stdout=subprocess.PIPE)
p5 = subprocess.Popen('cut -d \',\' -f2', stdin=p4.stdout, stdout=subprocess.PIPE)
p6 = subprocess.Popen('sed \'s/"//g\'', stdin=p5.stdout, stdout=subprocess.PIPE)
p7 = subprocess.Popen('sort', stdin=p6.stdout, stdout=subprocess.PIPE)
p8 = subprocess.Popen('sort', stdin=p8.stdout, stdout=subprocess.PIPE)
p9 = subprocess.Popen('uniq -c', stdin=p8.stdout, stdout=subprocess.PIPE)
p0 = subprocess.Popen('sort -nr', stdin=p9.stdout, stdout=subprocess.PIPE)
print(p01.communicate())

Tried now changing to lists.

p2 = subprocess.Popen('sed \'s/"/ /g\'', stdin=p1.stdout, stdout=subprocess.PIPE, shell=True)
p3 = subprocess.Popen(['grep','"shipOption"'], stdin=p2.stdout,stdout=subprocess.PIPE,shell = True)
p4 = subprocess.Popen(['grep','-v', '"getShipMethod"'], stdin=p3.stdout, stdout=subprocess.PIPE,shell = True)
p5 = subprocess.Popen(['cut','-d','\',\'', '-f2'], stdin=p4.stdout, stdout=subprocess.PIPE,shell = True)
p6 = subprocess.Popen(['sed','\'s/"//g\''],stdin=p5.stdout, stdout=subprocess.PIPE,shell = True)
p7 = subprocess.Popen(['sort'], stdin=p6.stdout, stdout=subprocess.PIPE,shell = True)
p8 = subprocess.Popen(['uniq', '-c'], stdin=p7.stdout, stdout=subprocess.PIPE,shell = True)
p9 = subprocess.Popen(['sort', '-nr'], stdin=p8.stdout, stdout=subprocess.PIPE,shell = True)
p0 = subprocess.Popen(['head', '-10'], stdin=p9.stdout, stdout=subprocess.PIPE,shell = True)```

New Error:

`usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]
usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-w | -d delim] [file ...]
(b'', None)
cat: stdin: Input/output error`

Gatha

You may need to be using double quote characters (") for the arguments being passing to sed — I don't think bash recognizes single quote (') characters they way you think. – Reproof

You either need to pass a [commandname, argument, ...] list, or a 'commandname arguments ... string and shell=True. In all but the first call, you're using the string form but without shell=True, so the entire string is being treated as a command name, not parsed into a command name and arguments. This looks like a duplicate of "Actual meaning of shell=True in subprocess". – Sinnard

Why would you do these things using subprocess instead of just writing code (or maybe writing a shell script)? Python can do all of those operations trivially, without imposing the overhead of a dozen external shell processes. – Mingmingche

The above part is just one function, I'm using other shell commands in other functions, i'm having a blocker around this step – Gatha

Answer 1

Your commands are still wrong. If you just want to run these commands like the shell does, the absolutely easiest way to do that is to ... use the shell.

result = subprocess.run('''
# useless cat, but bear with
cat output3.txt |
sed 's/"/ /g' |
grep "shipOption" |
grep -v "getShipMethod" |
cut -d ',' -f2 |
sed 's/"//g' |
sort |
uniq -c |
sort -nr |
head -10
    ''',
    # Probably add these too
    check=True,
    capture_output=True,
    # We are using the shell for piping etc
    shell=True)

If you want to remove the shell=True and manually run all these processes, you have to understand how the shell works. In particular, you need to fix the quoting so that the commands you run have the quotes which remain after the shell has processed the syntactic quotes.

p1 = subprocess.Popen(['cat', 'output3.txt'], stdout=subprocess.PIPE)  # still useless
p2 = subprocess.Popen(['sed','s/"/ /g'], stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(['grep', "shipOption"], stdin=p2.stdout,stdout=subprocess.PIPE)
p4 = subprocess.Popen(['grep', '-v', "getShipMethod"], stdin=p3.stdout, stdout=subprocess.PIPE)
p5 = subprocess.Popen(['cut', '-d', ',', '-f2'], stdin=p4.stdout, stdout=subprocess.PIPE)
p6 = subprocess.Popen(['sed', 's/"//g'],stdin=p5.stdout, stdout=subprocess.PIPE)
p7 = subprocess.Popen(['sort'], stdin=p6.stdout, stdout=subprocess.PIPE)
p8 = subprocess.Popen(['uniq', '-c'], stdin=p7.stdout, stdout=subprocess.PIPE)
p9 = subprocess.Popen(['sort', '-nr'], stdin=p8.stdout, stdout=subprocess.PIPE)
p0 = subprocess.Popen(['head', '-10'], stdin=p9.stdout, stdout=subprocess.PIPE)

Notice in particular how the arguments to sed and grep have their outer quotes removed, and how we removed shell=True everywhere. As a rule of thumb, if the first argument to Popen (or other subprocess methods) is a list, you should not use shell=True, and vice versa. (There are situations where you can pass a list to shell=True but ... let's not even begin to go there.)

All of this seems rather moot, though, since Python can eminently well do all of these things.

from collections import Counter

counts = Counter()
with open('output3.txt', 'r', encoding='utf-8') as lines:
    for line in lines:
        line = line.rstrip('\n').replace('"', ' ')
        if "shipOption" in line and "getShipMethod" not in line:
            field = line.split(',')[1].replace('"', '')
            counts[field] += 1
print(counts.most_common(10))

Probably you would want to put the rstrip and replace inside the if to avoid unnecessary work. The same refactoring could be done to the shell pipeline, of course.

Olette

See also useless use of cat – Olette

Answer 2

As per Gordon above - by default Popen() treats

sed 's/"/ /g'

as the name of a command to run rather than a command name plus one argument. So you need to do one of the following:

p2 = subprocess.Popen(['sed', 's/"/ /g'], stdin=p1.stdout, stdout=subprocess.PIPE)

or

p2 = subprocess.Popen('sed \'s/"/ /g\'', stdin=p1.stdout, stdout=subprocess.PIPE, shell=True)

The use of shell=True gets the Popen function to split the string into a list containing the command and arguments for you.

Also note, that p0 is your last result, but you call p01.communicate()

Equator

Answer 3

Suppose you use command 'python hello.py --main' to run your hello.py file with --main as an argument.

this will throw an error that no such or file or directory is found.

subprocess.run(["python", "hello.py --main"],stdout=subprocess.PIPE)

in subprocess you would write it as

subprocess.run(["python", "hello.py", "--main"],stdout=subprocess.PIPE)

you can put this in a variable and print the value returned from the script like this

data = subprocess.run(["python", "hello.py", "--main"],stdout=subprocess.PIPE)

print(data.stdout.decode('utf-8').strip())

Nonviolence