Pipe result from subprocess to unix sort

Question

I am calling a perl script on an external txt files from python, and printing the output to an outfile. But instead I want to pipe the output to unix's sort. Right now I am not piping, but are writing the output from the perl program first, then doing by combining my code under, and this stackoverflow answer.

import subprocess
import sys
import os

for file in os.listdir("."):

    with open(file + ".out", 'w') as outfile:
        p = subprocess.Popen(["perl", "pydyn.pl", file], stdout=outfile)
        p.wait()

Jacquelynnjacquenetta

You are using python incorrectly. If you are going to be calling a lot of outside processes (like sort), instead of using python modules, it's better to use bash, ipython or other shell. – Invigorate

you can also pipe the result p = subprocess.Popen("perl pydyn.pl %s | sort" % file, stdout=outfile,shell=True) but for this you're gonna have to make it shell=True which is not a good practice – Ailee

related: How do I use subprocess.Popen to connect multiple processes by pipes? – Clamant

Answer 1

Since you asked the question in python you can also pipe the result

p = subprocess.Popen("perl pydyn.pl %s | sort" % file, stdout=outfile,shell=True) 

but for this you're gonna have to make it shell=True which is not a good practice

Here's one way without making it shell=True

  p = subprocess.Popen(["perl", "pydyn.pl", file], stdout=subprocess.PIPE)
  output = subprocess.check_output(['sort'], stdin=p.stdout,stdout=outfile)
  p.wait()

Ailee

check_output is incorrect here; you can't change stdout. "%s" % file breaks if file contains space or $, etc e.g., it allows to execute arbitrary commands. You shoulld close p.stdout for proper cleanup. See my answer – Clamant

Answer 2

To emulate the shell pipeline:

#!/usr/bin/env python
import pipes
import subprocess

pipeline = "perl pydyn.pl {f} | sort >{f}.out".format(f=pipes.quote(filename))
subprocess.check_call(pipeline, shell=True)

without invoking the shell in Python:

#!/usr/bin/env python
from subprocess import Popen, PIPE

perl = Popen(['perl', 'pydyn.pl', filename], stdout=PIPE)
with perl.stdout, open(filename+'.out', 'wb', 0) as outfile:
    sort = Popen(['sort'], stdin=perl.stdout, stdout=outfile)
perl.wait() # wait for perl to finish
rc = sort.wait() # wait for `sort`, get exit status

Clamant

Answer 3

Just use bash. Using python just adds a level of complexity you don't need.

for file in $( ls); 
do 
    perl pydyn.pl $file | sort
done

Above is a quick and dirty example, a better alternative in terms of parsing is the following:

ls | while read file; do perl pydyn.pl "$file" | sort; done

Invigorate

it fails on the first file that has space in the name – Clamant

This could be due to a number of reasons. I added another example that will probably solve the issue. – Invigorate

it is incorrect. It should be read file, not read $file. bash skips over such errors silently. The Zen of Python: "Errors should never pass silently. Unless explicitly silenced." – Clamant