append subprocess.Popen output to file?

Question

I can successfully redirect my output to a file, however this appears to overwrite the file's existing data:

import subprocess
outfile = open('test','w') #same with "w" or "a" as opening mode
outfile.write('Hello')
subprocess.Popen('ls',stdout=outfile)

will remove the 'Hello' line from the file.

I guess a workaround is to store the output elsewhere as a string or something (it won't be too long), and append this manually with outfile.write(thestring) - but I was wondering if I am missing something within the module that facilitates this.

Serles

This is a deeper issue on Windows and append mode, see bugs.python.org/issue45237 for workarounds. – Cheri

Answer 1

You sure can append the output of subprocess.Popen to a file, and I make a daily use of it. Here's how I do it:

log = open('some file.txt', 'a')  # so that data written to it will be appended
c = subprocess.Popen(['dir', '/p'], stdout=log, stderr=log, shell=True)

(of course, this is a dummy example, I'm not using subprocess to list files...)

By the way, other objects behaving like file (with write() method in particular) could replace this log item, so you can buffer the output, and do whatever you want with it (write to file, display, etc) [but this seems not so easy, see my comment below].

Note: what may be misleading, is the fact that subprocess, for some reason I don't understand, will write before what you want to write. So, here's the way to use this:

log = open('some file.txt', 'a')
log.write('some text, as header of the file\n')
log.flush()  # <-- here's something not to forget!
c = subprocess.Popen(['dir', '/p'], stdout=log, stderr=log, shell=True)

So the hint is: do not forget to flush the output!

Homothallic

Note on the above post: after some trials, it seems that subprocess.Popen needs something really like a file for its stdout and stderrarguments, so giving to them a custom object with only a write method is not sufficient. I tried, but a fileno function is required, and I'm not good enough to emulate this on a class of my own. – Determinant

I have been attempting to use this method but I have discovered that for some reason, each time I run the external process with a file which I have definitely opened for appending, the output of the process writes from the beginning of the file and overwrites any information there. So, in order to use this solution, one must call log.seek( 0, os.SEEK_END ) immediately after opening the file. – Incus

Answer 2

Well the problem is if you want the header to be header, then you need to flush before the rest of the output is written to file :D

Naima

Answer 3

Are data in file really overwritten? On my Linux host I have the following behavior: 1) your code execution in the separate directory gets:

$ cat test
test
test.py
test.py~
Hello

2) if I add outfile.flush() after outfile.write('Hello'), results is slightly different:

$ cat test
Hello
test
test.py
test.py~

But output file has Hello in both cases. Without explicit flush() call stdout buffer will be flushed when python process is terminated. Where is the problem?

Morello

Answer 4

I faced the same problem and resolved it with the code below

      with open("outfile.log", 'w') as f: # use 'a' if you need to append
            subprocess.Popen(['ls', '-la'], shell=False,
                             stdout=f.fileno(),
                             stderr=f.fileno())

Greetings

Napper