Piping data from python to an external command

Question

I've read everything there is on subprocess.Popen but I think I'm missing something.

I need to be able to execute a unix program which reads a data stream from a list created in the python script and write the result of that program to a file. From the bash prompt I do this all the time with no problem but now I am trying to to this from within a python script which preprocesses some binary files and a lot of data before coming to this stage.

Lets look at a simple example not including all the preprocessing:

import sys
from pylab import *
from subprocess import *
from shlex import split

# some arbitrary x,y points
points = [(11,31),(13,33),(15,37),(16,35),(17,38),(18,39.55)]

commandline = 'my_unix_prog option1 option2 .... > outfile'
command = split(commandline)

process = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE)
print process.communicate(str(points))

The way this would be executed in bash is:

echo "11 31
      13 33
      15 37
      16 35
      17 38
      18 39.55" | my_unix_prog option1 option2 .... > outfile

The way the data is fed into the unix prog is important as well, I should be formatted in 2 columns separated by whitespace.

Any help is appreciated...

Ignatia

in the command you have the output redirected to outfile but then you try to read it into your program using communicate. Where do you want the output to go? – Hand

Using '>' is not the way to redirect output to a file, see this answer: #8902706 – Kibitz

Answer 1

SOLVED!

With the help of Dhara and xhainingx I was able to figure this out:

import sys
from pylab import *
from subprocess import *
from shlex import split

# some arbitrary x,y points
points = [(11,31),(13,33),(15,37),(16,35),(17,38),(18,39.55)]

commandline = 'my_unix_prog option1 option2 ....'
command = split(commandline)

process = Popen(command, stdin=PIPE, stdout=open('outfile', 'w'), stderr=PIPE)
for p in points:
    process.stdin.write(str(p[0]) + ' ' + str(p[1]) + '\n')

print process.communicate()

This works very well, thanks.

Ignatia

I'd also recommend that you create the file object outside the call the Popen so you can later call .close() on it when you and your subprocess are through with it – Hand

Answer 2

how about something like

for p in points:
    process.stdin.write(str(p[0]) + ' ' + str(p[1]) + '\n')

print process.communicate()

Hand

Yes, you are correct about that (both of you) but I think my problems begin before that: I use the '>' redirection but this is shell redirection and is not valid. – Ignatia

@Ignatia you can use the shell=True argument in your Popen constructor to allow the shell redirection, but then communicate won't return anything for stdout. Another option is to open a file f=open('outfile', 'w') and then instead of PIPE use stdout=f. I'm not sure what the problem is now – Hand

Answer 3

You need to format your input to communicate correctly.

str will keep the special characters when you print your list of tuples which isn't what you want.

>>> print str([(1,2), (3,4)]) 
[(1,2), (3,4)]

Try this:

print process.communicate("\n".join(["%s %s"%(x[0], x[1]) for x in points])

Remanent