How to redirect 'print' output to a file?

Question

I want to redirect the print to a .txt file using Python. I have a for loop, which will print the output for each of my .bam file while I want to redirect all output to one file. So I tried to put:

f = open('output.txt','w')
sys.stdout = f

at the beginning of my script. However I get nothing in the .txt file. My script is:

#!/usr/bin/python

import os,sys
import subprocess
import glob
from os import path

f = open('output.txt','w')
sys.stdout = f

path= '/home/xxx/nearline/bamfiles'
bamfiles = glob.glob(path + '/*.bam')

for bamfile in bamfiles:
    filename = bamfile.split('/')[-1]
    print 'Filename:', filename
    samtoolsin = subprocess.Popen(["/share/bin/samtools/samtools","view",bamfile],
                                  stdout=subprocess.PIPE,bufsize=1)
    linelist= samtoolsin.stdout.readlines()
    print 'Readlines finished!'

So what's the problem? Any other way besides this sys.stdout?

I need my result look like:

Filename: ERR001268.bam
Readlines finished!
Mean: 233
SD: 10
Interval is: (213, 252)

Carrack

Why not use f.write(data)? – Dimitrovo

yeah, but I have several data for each bam file (mean, SD,interval...), how can I put these data one by one? – Carrack

@Eran Zimmerman: f.write(line) does not add a line break to the data. – Burundi

You're right, my bad. Could always f.write(line+'\n'), however.. – Dimitrovo

you don't see anything because you aren't writing enough, in which case you need to call .flush() on the file. And for some editors like kate, even with auto-reload enabled, for some unknown reason, you have to re-open the file too to see the changes. – Volt

Answer 1

The most obvious way to do this would be to print to a file object:

with open('out.txt', 'w') as f:
    print('Filename:', filename, file=f)  # Python 3.x
    # print >> f, 'Filename:', filename   # Python 2.x

However, redirecting stdout also works for me. It is probably fine for a one-off script such as this:

import sys

orig_stdout = sys.stdout
f = open('out.txt', 'w')
sys.stdout = f

for i in range(2):
    print('i = ', i)

sys.stdout = orig_stdout
f.close()

Since Python 3.4 there's a simple context manager available to do this in the standard library:

from contextlib import redirect_stdout

with open('out.txt', 'w') as f:
    with redirect_stdout(f):
        print('data')

Redirecting externally from the shell itself is another option, and often preferable:

./script.py > out.txt

Other questions:

What is the first filename in your script? I don't see it initialized.

My first guess is that glob doesn't find any bamfiles, and therefore the for loop doesn't run. Check that the folder exists, and print out bamfiles in your script.

Also, use os.path.join and os.path.basename to manipulate paths and filenames.

Bossy

Line 8 of your code uses a variable named filename, but it hasn't been created yet. Later in the loop you use it again, but not relevant. – Bossy

Bad practice to change sys.stdout if you don't need to. – Margarettmargaretta

@my I'm not convinced it is bad for a simple script like this. – Bossy

+1 Haha well you can have my upvote because it's the right way to do it if you absolutely must do it the wrong way... But I still say you should do it with regular file output. – Margarettmargaretta

instead of orig_stdout = sys.stdout then sys.stdout = orig_stdout you can just do sys.stdout = sys.__stdout__ – Rhodesia

That's not a good habit to get into; if stdout has already been redirected, the association will be lost. – Bossy

open() not file() – Transposal

How to redirect and print the output on the console ? Seems the "print()" in Python cannot be shown when the stdrr is redirected ? – Standpoint

@exteralvictor Sounds like you are looking for the tee file.out command. If on Windows, perhaps duplicate the print() call, one with file=f, one without. A simple wrapping function could automate. – Bossy

Why it's a bad habit? Actually that's the easiest way to pipe the prints of an actively developed script to a logfile. – Quiver

How I append output to targer file with this method "./script.py > out.txt" ? I don't want to owerrite the file. – Thyroid

@UğurGürkanTosun , ./script.py >> out.txt See the double arrow? – Bossy

@exteral: you have to flush the file (call .flush()) to get the output to show – Volt

Answer 2

You can redirect print with the file argument (in Python 2 there was the >> operator instead).

f = open(filename,'w')
print('whatever', file=f) # Python 3.x
print >>f, 'whatever'     # Python 2.x

In most cases, you're better off just writing to the file normally.

f.write('whatever')

or, if you have several items you want to write with spaces between, like print:

f.write(' '.join(('whatever', str(var2), 'etc')))

Castellated

If there are a lot of output statements these can get old fast. The posters original idea is valid; there is something else wrong with the script. – Bossy

Poster's original idea is absolutely invalid. There's no reason to redirect stdout here, since he already gets the data into a variable. – Margarettmargaretta

I think he meant "technically valid", in that you can, in fact, redirect sys.stdout, not that it was a good idea. – Castellated

Answer 3

Python 2 or Python 3 API reference:

print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)

The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used. Since printed arguments are converted to text strings, print() cannot be used with binary mode file objects. For these, use file.write(...) instead.

Since file object normally contains write() method, all you need to do is to pass a file object into its argument.

Write/Overwrite to File

with open('file.txt', 'w') as f:
    print('hello world', file=f)

Write/Append to File

with open('file.txt', 'a') as f:
    print('hello world', file=f)

Stab

I just confused why some of those earlier answers were to monkey patch the global sys.stdout :( – Stab

Say, you have some legacy codes (2.7) with print statements (instead of logs) all over the file. Now, you want those to be logged into file for debug/investigation purpose. In this case, modifying the global stdout is preferable to modifying each and every print statement. Just my opinion. – Miliary

Answer 4

Don't use print, use logging

You can change sys.stdout to point to a file, but this is a pretty clunky and inflexible way to handle this problem. Instead of using print, use the logging module.

With logging, you can print just like you would to stdout, or you can also write the output to a file. You can even use the different message levels (critical, error, warning, info, debug) to, for example, only print major issues to the console, but still log minor code actions to a file.

A simple example

Import logging, get the logger, and set the processing level:

import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG) # process everything, even if everything isn't printed

If you want to print to stdout:

ch = logging.StreamHandler()
ch.setLevel(logging.INFO) # or any other level
logger.addHandler(ch)

If you want to also write to a file (if you only want to write to a file skip the last section):

fh = logging.FileHandler('myLog.log')
fh.setLevel(logging.DEBUG) # or any level you want
logger.addHandler(fh)

Then, wherever you would use print use one of the logger methods:

# print(foo)
logger.debug(foo)

# print('finishing processing')
logger.info('finishing processing')

# print('Something may be wrong')
logger.warning('Something may be wrong')

# print('Something is going really bad')
logger.error('Something is going really bad')

To learn more about using more advanced logging features, read the excellent logging tutorial in the Python docs.

Dukedom

Hi, I want to use this logging for writing the console data to the log file with the time like at which time that data is taken. But i am not able to understand the logging function or library properly. Can you help me with this – Scammon

@Scammon Read through the Python docs' logging tutorial and check out examples in other questions on Stack Overflow (there are a lot of them). If you are still unable to get it to work, ask a new question. – Dukedom

To print multiple variables like 'a' and 'b', for print function you can just do print(a, b). In logging, you have to do logger.info('%s %s' %(a, b)), or logger.info(f'{a}, {b}'), both are much harder. – Oao

Answer 5

This works perfectly:

import sys
sys.stdout=open("test.txt","w")
print ("hello")
sys.stdout.close()

Now the hello will be written to the test.txt file. Make sure to close the stdout with a close, without it the content will not be save in the file

Denadenae

but even if we perform sys.stdout.close() , if you type anything in python shell it will show error as ValueError: I/O operation on closed file. imgur.com/a/xby9P. Best way to handle this is follow what @Gringo Suave posted – Sardinia

Answer 6

The easiest solution isn't through python; its through the shell. From the first line of your file (#!/usr/bin/python) I'm guessing you're on a UNIX system. Just use print statements like you normally would, and don't open the file at all in your script. When you go to run the file, instead of

./script.py

to run the file, use

./script.py > <filename>

where you replace <filename> with the name of the file you want the output to go in to. The > token tells (most) shells to set stdout to the file described by the following token.

One important thing that needs to be mentioned here is that "script.py" needs to be made executable for ./script.py to run.

So before running ./script.py,execute this command

chmod a+x script.py (make the script executable for all users)

Cychosz

./script.py > <filename> 2>&1 You need to capture stderr as well. 2>&1 will do that – Micrometeorite

@Micrometeorite Why? The question specifically wants to pipe the output of print to a file. It would be reasonable to expect stdout (stack traces and the like) to still print to the terminal. – Cychosz

He said it wasn't working, mine wasn't working either. I later discovered that this app I'm working on was configured to direct everything to stderr...idk why. – Micrometeorite

Answer 7

If you are using Linux I suggest you to use the tee command. The implementation goes like this:

python python_file.py | tee any_file_name.txt

If you don't want to change anything in the code, I think this might be the best possible solution. You can also implement logger but you need do some changes in the code.

Leprechaun

Answer 8

You may not like this answer, but I think it's the RIGHT one. Don't change your stdout destination unless it's absolutely necessary (maybe you're using a library that only outputs to stdout??? clearly not the case here).

I think as a good habit you should prepare your data ahead of time as a string, then open your file and write the whole thing at once. This is because input/output operations are the longer you have a file handle open, the more likely an error is to occur with this file (file lock error, i/o error, etc). Just doing it all in one operation leaves no question for when it might have gone wrong.

Here's an example:

out_lines = []
for bamfile in bamfiles:
    filename = bamfile.split('/')[-1]
    out_lines.append('Filename: %s' % filename)
    samtoolsin = subprocess.Popen(["/share/bin/samtools/samtools","view",bamfile],
                                  stdout=subprocess.PIPE,bufsize=1)
    linelist= samtoolsin.stdout.readlines()
    print 'Readlines finished!'
    out_lines.extend(linelist)
    out_lines.append('\n')

And then when you're all done collecting your "data lines" one line per list item, you can join them with some '\n' characters to make the whole thing outputtable; maybe even wrap your output statement in a with block, for additional safety (will automatically close your output handle even if something goes wrong):

out_string = '\n'.join(out_lines)
out_filename = 'myfile.txt'
with open(out_filename, 'w') as outf:
    outf.write(out_string)
print "YAY MY STDOUT IS UNTAINTED!!!"

However if you have lots of data to write, you could write it one piece at a time. I don't think it's relevant to your application but here's the alternative:

out_filename = 'myfile.txt'
outf = open(out_filename, 'w')
for bamfile in bamfiles:
    filename = bamfile.split('/')[-1]
    outf.write('Filename: %s' % filename)
    samtoolsin = subprocess.Popen(["/share/bin/samtools/samtools","view",bamfile],
                                  stdout=subprocess.PIPE,bufsize=1)
    mydata = samtoolsin.stdout.read()
    outf.write(mydata)
outf.close()

Margarettmargaretta

With disk caching performance of the original should be acceptable. This solution however has the drawback of ballooning the memory requirements if there were a lot of output. Though probably nothing to worry about here, it is generally a good idea to avoid this if possible. Same idea as using xrange (py3 range) instead of range, etc. – Bossy

@Gringo: He didn't specify this requirement. Rarely do I ever write enough data to a file that this would be relevant. This is not the same idea as xrange because xrange doesn't deal with file i/o. Disk caching might help but it's still a bad practice to keep a file handle open for a large body of code. – Margarettmargaretta

Your comment contradicts itself. To be honest the performance aspect of both approaches is irrelevant for non-huge amounts of data. xrange certainly is similar, it works on one piece at a time instead of all at once in memory. Perhaps a generator vs list is a better example though. – Bossy

@Gringo: I fail to see how my comment contradicts itself. Maybe the performance aspect isn't relevant, keeping a file handle open for an extended period always increases the risk of error. In programming file i/o is always inherently more risky than doing something within your own program, because it means you have to reach out through the OS and mess around with file locks. The shorter you have a file open for, the better, simply because you don't control the file system from your code. xrange is different because it has nothing to do with file i/o, and FYI I rarely use xrange either; cheers – Margarettmargaretta

@Gringo: Took out the bit about performance, made it all about error-carefulness. Happy? :) – Margarettmargaretta

@Gringo: That's not true at all. You can do a lot of computation, taking a long time in your code, without writing megabytes of data. I understand why you think xrange applies, but it has nothing to do with my reasoning. It's a better coding practice to not have a bunch of open file handles hanging around primarily due to file locks. Your comment about xrange is irrelevant because there was never any concern for memory efficiency, and there rarely is! In any case you don't need to touch sys.stdout at all here, should stick with regular output variables, so your answer IS wrong. – Margarettmargaretta

@GringoSuave let us continue this discussion in chat – Margarettmargaretta

@Gringo: I appreciate your criticism and enjoyed the heated debate. Even though we disagreed on some points I still respect your views as it's clear you have a good reason for taking your stance. Thanks for ending it reasonably and have a very good night. :P – Margarettmargaretta

Answer 9

If redirecting stdout works for your problem, Gringo Suave's answer is a good demonstration for how to do it.

To make it even easier, I made a version utilizing contextmanagers for a succinct generalized calling syntax using the with statement:

from contextlib import contextmanager
import sys

@contextmanager
def redirected_stdout(outstream):
    orig_stdout = sys.stdout
    try:
        sys.stdout = outstream
        yield
    finally:
        sys.stdout = orig_stdout

To use it, you just do the following (derived from Suave's example):

with open('out.txt', 'w') as outfile:
    with redirected_stdout(outfile):
        for i in range(2):
            print('i =', i)

It's useful for selectively redirecting print when a module uses it in a way you don't like. The only disadvantage (and this is the dealbreaker for many situations) is that it doesn't work if one wants multiple threads with different values of stdout, but that requires a better, more generalized method: indirect module access. You can see implementations of that in other answers to this question.

Epi

Good idea. This context manager exists in the contextlib module however, docs say since Python 3.4. – Bossy

@GringoSuave Yep, the contextmanager function should be imported to use as a decorator; hence why I put from contextlib import contextmanager at the top of the code block. – Epi

Answer 10

Here's another method I've used for printing to a file/log... Modify the built-in print function so that it logs to a file in the temp directory with the current time stamp, as well as print to stdout. The only real advantage to doing this within a script is not having to go and modify existing print statements.

print('test')

test

Copy original print function to new variable

og_print = print
og_print('test2')

test2

Overwrite existing print function

def print(*msg):
    '''print and log!'''
    # import datetime for timestamps
    import datetime as dt
    # convert input arguments to strings for concatenation
    message = []
    for m in msg:
        message.append(str(m))
    message = ' '.join(message)
    # append to the log file
    with open('/tmp/test.log','a') as log:
        log.write(f'{dt.datetime.now()} | {message}\n')
    # print the message using the copy of the original print function to stdout
    og_print(message)

print('test3')

test3

display file

cat /tmp/test.log

2022-01-25 10:19:11.045062 | test3

remove file

rm /tmp/test.log

Notability

Answer 11

I am able to crack this using the following method. It will use this print function instead of builtin print function and save the content to a file.

from __future__ import print_function
import builtins as __builtin__

log = open("log.txt", "a")

def print(*args):
    newLine = ""
    for item in args:
        newLine = newLine + str(item) + " "
    newLine = (
        newLine
        + """
"""
    )
    log.write(newLine)
    log.flush()
    __builtin__.print(*args)
    return

Individual

Haha this is a nice trick, do you use this kind of trick all the time? :) – Rosenberger

Yeah, especially in winter season. lol. ;) – Individual

Answer 12

Changing the value of sys.stdout does change the destination of all calls to print. If you use an alternative way to change the destination of print, you will get the same result.

Your bug is somewhere else:

• it could be in the code you removed for your question (where does filename come from for the call to open?)
• it could also be that you are not waiting for data to be flushed: if you print on a terminal, data is flushed after every new line, but if you print to a file, it's only flushed when the stdout buffer is full (4096 bytes on most systems).

Concordat

Answer 13

In python 3, you can reassign print:

#!/usr/bin/python3

def other_fn():
    #This will use the print function that's active when the function is called
    print("Printing from function")

file_name = "test.txt"
with open(file_name, "w+") as f_out:
    py_print = print #Need to use this to restore builtin print later, and to not induce recursion
   
    print = lambda out_str : py_print(out_str, file=f_out)
    
    #If you'd like, for completeness, you can include args+kwargs
    print = lambda *args, **kwargs : py_print(*args, file=f_out, **kwargs)
    
    print("Writing to %s" %(file_name))

    other_fn()  #Writes to file

    #Must restore builtin print, or you'll get 'I/O operation on closed file'
    #If you attempt to print after this block
    print = py_print

print("Printing to stdout")
other_fn() #Writes to console/stdout

Note that the print from other_fn only switches outputs because print is being reassigned in the global scope. If we assign print within a function, the print in other_fn is normally not affected. We can use the global keyword if we want to affect all print calls:

import builtins

def other_fn():
    #This will use the print function that's active when the function is called
    print("Printing from function")

def main():
    global print #Without this, other_fn will use builtins.print
    file_name = "test.txt"
    with open(file_name, "w+") as f_out:

        print = lambda *args, **kwargs : builtins.print(*args, file=f_out, **kwargs)

        print("Writing to %s" %(file_name))

        other_fn()  #Writes to file

        #Must restore builtin print, or you'll get 'I/O operation on closed file'
        #If you attempt to print after this block
        print = builtins.print

    print("Printing to stdout")
    other_fn() #Writes to console/stdout

Personally, I'd prefer sidestepping the requirement to use the print function by baking the output file descriptor into a new function:

file_name = "myoutput.txt"
with open(file_name, "w+") as outfile:
    fprint = lambda pstring : print(pstring, file=outfile)
    print("Writing to stdout")
    fprint("Writing to %s" % (file_name))

Rubidium

Answer 14

Something that I have used in the past to output some dictionaries is the following:

# sample dictionary
the_dict = {'a': 'no', 'c': 'yes', 'b': 'try again'}

# path to output to
dict_path = "D:/path.txt"

# script to output file
with open(dict_path, "w") as f:
    for idx, data in the_dict.items():
        print(idx, data, file=f)

The outputted file will look something like below:

a no
c yes
b try again

Owensby

Answer 15

Something to extend print function for loops

x = 0
while x <=5:
    x = x + 1
    with open('outputEis.txt', 'a') as f:
        print(x, file=f)
    f.close()

Allure

no need to use while and no need to close the file when using with – Strangeness