Redirect stdout to a file in Python? [duplicate]
Asked Answered
H

15

439

How do I redirect stdout to an arbitrary file in Python?

When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.

I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I've tested on, but I also didn't have time yet to test on a web application yet.

Hawker answered 13/1, 2011 at 0:51 Comment(11)
That's not really a python thing, it's a shell function. Just run your script like script.p > filePrisca
I currently solve the problem using nohup, but I thought there might be something more clever...Hawker
@foxbunny: nohup? Why simply someprocess | python script.py? Why involve nohup?Aniela
@S.Lott: Why someprocess | python script.py?Hawker
@foxbunny: My mistake. Your question is badly worded. I thought you wanted to redirect stdout of some process into a Python program. It appears, from the accepted answer, that you're doing something different. I would still like to know how you're using nohup. And why.Aniela
@S.Lott: I accepted the answer because I think it'd work. But I haven't implemented it yet. I use nohup to avoid having to worry about stdout in case I start the script from a terminal, background it, and then exit the terminal like so: nohup python script.py > logfile & I still use that technique because simply assigning a file handle to sys.stdout didn't do the trick for some stuff like the cherrypy server. They still happily write to stdout, but in some cases that had fatal consequences.Hawker
@foxbunny: "assigning a file handle to sys.stdout didn't do the trick for some stuff like the cherrypy server" What? That doesn't make sense. It shouldn't work. It's a web server not a file processing application. This is a very confusing question. Can you update the question to clarify what you're talking about?Aniela
@S.Lott: After a few more tests, I corrected some errors I had in the code, and now (contrary to what you claim) stdout is universally redirected to a file as per the solutions in the two answers below. If you still think it shouldn't work, you should try it yourself. If you still don't understand the question itself, though, unfortunately that's not my concern. Again, look at the two provided answers and you should be able to figure it out.Hawker
Rewrite the print statements to apply the logging module from the stdlib. Then you can redirect output everywhere, have control over how much output you want etc. In most cases production code should not print but log.Myotonia
Perhaps a better solution for this problem is the screen command, which will save your bash session and allow you to access it from different runs.Aspirant
Special case of suppressing the output (there is a presumably faster solution by redirecting to None instead of /dev/null): Redirecting stdout to "nothing" in python - Stack OverflowBlen
S
570

If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick:

# for python3
import sys
with open('file', 'w') as sys.stdout:
    print('test')

A far more common method is to use shell redirection when executing (same on Windows and Linux):

$ python3 foo.py > file
Semiprofessional answered 13/1, 2011 at 0:53 Comment(16)
If you're on Windows watch out for Windows bug - Cannot redirect output when I run Python script on Windows using just script's namePotentiality
It doesn't work with from sys import stdout, maybe because it creates a local copy. Also you can use it with with, e.g. with open('file', 'w') as sys.stdout: functionThatPrints(). You can now implement functionThatPrints() using normal print statements.Fairfax
It's best to keep a local copy, stdout = sys.stdout so you can put it back when you're done, sys.stdout = stdout. That way if you're being called from a function that uses print you don't screw them up.Fairfax
+1 for ./foo > file. Note: if you want only temporarily redirect the stdout then see @Gerli's answer and my answer that shows both how to redirect at sys.stdout and file descriptors levelAmusement
Using python foo.py > file has a potential problem, see my question hereEnvy
Is this supposed to give a continuous output? I would like to monitor the output, e.g. via tail -f logfile... But when I try both your suggestions it seems like the output is written only when the job is finished.Beeck
Sorry for the disturbance - one can use sys.stdout = open('file', 'w', 0) for continuous, i.e. unbuffered, output.Beeck
@Jan: buffering=0 disables buffering (it may negatively affect performance (10-100 times)). buffering=1 enables line buffering so that you could use tail -f for a line-oriented output.Amusement
@Fairfax or you can use sys.stdout = sys.__stdout__ to get it back.Braden
@mgold, how was it even supposed to work with from sys import stdout? What should be the code to redirect stdout if sys is not available (not imported)??Mindoro
do we need to close the file?Oleaceous
I just get a blank file (and still get output printed to the console) when I'm an calling an SDK - if I use the simple example it works - so could the SDK be "hiding" stdout?Livialivid
Reassigning built-in module constants is bad practice.Pood
@clemtoy: Had the same task, but it seems that your suggestion did not work (IDLE/Win10), the storing solution worked though. (Also, resetting stdout closes the output file.)Oriane
for a short non-web application script I was just able to redirect on command line without any alterations to code: python foo.py > file.txtOrnithology
Is it possible to add a timestamp for each line in stdout?Between
A
278

There is contextlib.redirect_stdout() function in Python 3.4+:

from contextlib import redirect_stdout

with open('help.txt', 'w') as f:
    with redirect_stdout(f):
        print('it now prints to `help.text`')

It is similar to:

import sys
from contextlib import contextmanager

@contextmanager
def redirect_stdout(new_target):
    old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
    try:
        yield new_target # run some code with the replaced stdout
    finally:
        sys.stdout = old_target # restore to the previous value

that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.

It doesn't redirect the stdout at the file descriptors level e.g.:

import os
from contextlib import redirect_stdout

stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
    print('redirected to a file')
    os.write(stdout_fd, b'not redirected')
    os.system('echo this also is not redirected')

b'not redirected' and 'echo this also is not redirected' are not redirected to the output.txt file.

To redirect at the file descriptor level, os.dup2() could be used:

import os
import sys
from contextlib import contextmanager

def fileno(file_or_fd):
    fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
    if not isinstance(fd, int):
        raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
    return fd

@contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
    if stdout is None:
       stdout = sys.stdout

    stdout_fd = fileno(stdout)
    # copy stdout_fd before it is overwritten
    #NOTE: `copied` is inheritable on Windows when duplicating a standard stream
    with os.fdopen(os.dup(stdout_fd), 'wb') as copied: 
        stdout.flush()  # flush library buffers that dup2 knows nothing about
        try:
            os.dup2(fileno(to), stdout_fd)  # $ exec >&to
        except ValueError:  # filename
            with open(to, 'wb') as to_file:
                os.dup2(to_file.fileno(), stdout_fd)  # $ exec > to
        try:
            yield stdout # allow code to be run with the redirected stdout
        finally:
            # restore stdout to its previous value
            #NOTE: dup2 makes stdout_fd inheritable unconditionally
            stdout.flush()
            os.dup2(copied.fileno(), stdout_fd)  # $ exec >&copied

The same example works now if stdout_redirected() is used instead of redirect_stdout():

import os
import sys

stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
    print('redirected to a file')
    os.write(stdout_fd, b'it is redirected now\n')
    os.system('echo this is also redirected')
print('this is goes back to stdout')

The output that previously was printed on stdout now goes to output.txt as long as stdout_redirected() context manager is active.

Note: stdout.flush() does not flush C stdio buffers on Python 3 where I/O is implemented directly on read()/write() system calls. To flush all open C stdio output streams, you could call libc.fflush(None) explicitly if some C extension uses stdio-based I/O:

try:
    import ctypes
    from ctypes.util import find_library
except ImportError:
    libc = None
else:
    try:
        libc = ctypes.cdll.msvcrt # Windows
    except OSError:
        libc = ctypes.cdll.LoadLibrary(find_library('c'))

def flush(stream):
    try:
        libc.fflush(None)
        stream.flush()
    except (AttributeError, ValueError, IOError):
        pass # unsupported

You could use stdout parameter to redirect other streams, not only sys.stdout e.g., to merge sys.stderr and sys.stdout:

def merged_stderr_stdout():  # $ exec 2>&1
    return stdout_redirected(to=sys.stdout, stdout=sys.stderr)

Example:

from __future__ import print_function
import sys

with merged_stderr_stdout():
     print('this is printed on stdout')
     print('this is also printed on stdout', file=sys.stderr)

Note: stdout_redirected() mixes buffered I/O (sys.stdout usually) and unbuffered I/O (operations on file descriptors directly). Beware, there could be buffering issues.

To answer, your edit: you could use python-daemon to daemonize your script and use logging module (as @erikb85 suggested) instead of print statements and merely redirecting stdout for your long-running Python script that you run using nohup now.

Amusement answered 16/3, 2014 at 7:39 Comment(10)
stdout_redirected is helpful. Be aware this doesn't work inside doctests, since the special SpoofOut handler doctest uses to replace sys.stdout doesn't have a fileno attribute.Loftus
@ChrisJohnson: If it doesn't raise ValueError("Expected a file (`.fileno()`) or a file descriptor") then it is a bug. Are you sure it doesn't raise it?Amusement
It does raise that error, which is what make it not usable within a doctest. To use your function within a doctest, it appears necessary to specify doctest.sys.__stdout__ where we would normally use sys.stdout. This isn't a problem with your function, just an accommodation required for doctest since it replaces stdout with an object that doesn't have all the attributes a true file would.Loftus
stdout_redirected() has stdout parameter, you could set it to sys.__stdout__ if you want to redirect the original python stdout (that should have a valid .fileno() in most cases). It does nothing for the current sys.stdout if they are different. Don't use doctest.sys; it is available by accident.Amusement
This really works well, i.e. redirect stdout and stderr to a fd: with stdout_redirected(to=fd): with merged_stderr_stdout(): print('...'); print('...', file=sys.stderr)Orland
Can this work with redirecting outputs from C++ to log files created with the Python logger module?Galicia
@BigBrownBear00: if you know what file descriptors are used in C++ code then they can be redirected to the desired files (though it might not be good idea to redirect into the files used by the logging module: it would to mix buffered/unbuffered I/O)Amusement
This can now be done in a few lines by using the wurlitzer module (sys_pipes context manager) from PyPI. See my answer below for details or read the wurlitzer documentation.Lacto
@Joooeey: the blog post linked from the wurlitzer pypi page says that it was inspired by the predecessor of the current answer. It is great that the functionality is available as library but there are many subtleties and tradeoffs with this kind of code -- depending your use-case you might need make different choices.Amusement
Maybe not relevant, but I don't think the os.dup2() example affects subprocesses, at least not those created with subprocess.Popen() (or at least not some, I'm not sure if mine is using spawn or fork if that matters). I have a test suite that specifies stdout=subprocess.DEVNULL to Popen(), but doesn't specify anything for stderr, so garbage gets printed during tests. I'll just specify stderr in this case, but it would be nice to prevent garbage from creeping in in the future too.Wilmott
T
103

you can try this too much better

import sys

class Logger(object):
    def __init__(self, filename="Default.log"):
        self.terminal = sys.stdout
        self.log = open(filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)

sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt
Tomika answered 6/5, 2011 at 20:49 Comment(9)
Any suggestions for piping to logger or syslog?Benison
If you want to edit a file this isn't very useful. Anyway +1 for the nice trickVirginavirginal
This will have consequences for code which assumes sys.stdout is a full fledged file object with methods such as fileno() (which includes code in the python standard library). I would add a __getattr__(self, attr) method to that which defers attribute lookup to self.terminal. def __getattr__(self, attr): return getattr(self.terminal, attr)Acidfast
You have to add def flush(self): method as well to class Logger.Stereo
This helped me setup logging to both stdout and a file: docs.python.org/3/howto/logging-cookbook.htmlTrapan
@Stereo what would the flush method do?Reardon
@Reardon the flush will end up accumulating char in the buffer, print out and clean the buffer.Stereo
@Stereo but what actually goes in the method that you create?Reardon
Is there some way to implement this on Threads? cause on change sys.stdout on a thread it will change for all threads(tried and tested). I readed they suggest to use multiprocessing to avoid (cause it wont share stdout descriptor), but then get the common error Cant' Pickle bla bla bla ... Can sys.stdout be cloned and managed on each trhead?Usury
C
33

The other answers didn't cover the case where you want forked processes to share your new stdout.

To do that:

from os import open, close, dup, O_WRONLY

old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1

..... do stuff and then restore

close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs
Contemplative answered 24/7, 2012 at 14:24 Comment(6)
one needs to replace the 'w' attribute with, os.O_WRONLY|os.O_CREATE ... can't send strings into the "os" commands!Beyrouth
Insert a sys.stdout.flush() before the close(1) statement to make sure the redirect 'file' file gets the output. Also, you can use a tempfile.mkstemp() file in place of 'file'. And be careful you don't have other threads running that can steal the os's first file handle after the os.close(1) but before the 'file' is opened to use the handle.Avocation
its os.O_WRONLY | os.O_CREAT ... there is no E on there.Lubow
+1. You could also use os.dup2() and wrap it into a context manager as shown in my answerAmusement
@Ch'marr It's O_CREAT, not O_CREATE.Insincerity
The dup function makes the file descriptor non inheritable. This leads to failures when you run subprocess writing to stdout after the redirection is finished.Bellow
A
31

Quoted from PEP 343 -- The "with" Statement (added import statement):

Redirect stdout temporarily:

import sys
from contextlib import contextmanager
@contextmanager
def stdout_redirected(new_stdout):
    save_stdout = sys.stdout
    sys.stdout = new_stdout
    try:
        yield None
    finally:
        sys.stdout = save_stdout

Used as follows:

with open(filename, "w") as f:
    with stdout_redirected(f):
        print "Hello world"

This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example in scripts) it is a popular way of doing things.

Adellaadelle answered 5/2, 2013 at 12:9 Comment(2)
+1. Note: it doesn't work for subprocesses e.g., os.system('echo not redirected'). My answer shows how to redirect such outputAmusement
starting from Python 3.4 there is redirect_stdout in contextlibRhesus
L
14
import sys
sys.stdout = open('stdout.txt', 'w')
Lardon answered 13/1, 2011 at 0:52 Comment(0)
S
7

Here is a variation of Yuda Prawira answer:

  • implement flush() and all the file attributes
  • write it as a contextmanager
  • capture stderr also

.

import contextlib, sys

@contextlib.contextmanager
def log_print(file):
    # capture all outputs to a log file while still printing it
    class Logger:
        def __init__(self, file):
            self.terminal = sys.stdout
            self.log = file

        def write(self, message):
            self.terminal.write(message)
            self.log.write(message)

        def __getattr__(self, attr):
            return getattr(self.terminal, attr)

    logger = Logger(file)

    _stdout = sys.stdout
    _stderr = sys.stderr
    sys.stdout = logger
    sys.stderr = logger
    try:
        yield logger.log
    finally:
        sys.stdout = _stdout
        sys.stderr = _stderr


with log_print(open('mylogfile.log', 'w')):
    print('hello world')
    print('hello world on stderr', file=sys.stderr)

# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
#   ....
#   print('[captured output]', log.getvalue())
Sancha answered 19/2, 2018 at 16:48 Comment(0)
S
5

You need a terminal multiplexer like either tmux or GNU screen

I'm surprised that a small comment by Ryan Amos' to the original question is the only mention of a solution far preferable to all the others on offer, no matter how clever the python trickery may be and how many upvotes they've received. Further to Ryan's comment, tmux is a nice alternative to GNU screen.

But the principle is the same: if you ever find yourself wanting to leave a terminal job running while you log-out, head to the cafe for a sandwich, pop to the bathroom, go home (etc) and then later, reconnect to your terminal session from anywhere or any computer as though you'd never been away, terminal multiplexers are the answer. Think of them as VNC or remote desktop for terminal sessions. Anything else is a workaround. As a bonus, when the boss and/or partner comes in and you inadvertently ctrl-w / cmd-w your terminal window instead of your browser window with its dodgy content, you won't have lost the last 18 hours-worth of processing!

Sicard answered 17/5, 2015 at 14:13 Comment(1)
while it is a good answer for the part of the question appeared after the edit; it does not answer the question in the title (most people come here from google for the title)Amusement
R
3

Based on this answer: https://mcmap.net/q/25573/-redirect-stdout-to-a-file-in-python-duplicate, here is another way I figured out which I use in one of my projects. For whatever you replace sys.stderr or sys.stdout with, you have to make sure that the replacement complies with file interface, especially if this is something you are doing because stderr/stdout are used in some other library that is not under your control. That library may be using other methods of file object.

Check out this way where I still let everything go do stderr/stdout (or any file for that matter) and also send the message to a log file using Python's logging facility (but you can really do anything with this):

class FileToLogInterface(file):
    '''
    Interface to make sure that everytime anything is written to stderr, it is
    also forwarded to a file.
    '''

    def __init__(self, *args, **kwargs):
        if 'cfg' not in kwargs:
            raise TypeError('argument cfg is required.')
        else:
            if not isinstance(kwargs['cfg'], config.Config):
                raise TypeError(
                    'argument cfg should be a valid '
                    'PostSegmentation configuration object i.e. '
                    'postsegmentation.config.Config')
        self._cfg = kwargs['cfg']
        kwargs.pop('cfg')

        self._logger = logging.getlogger('access_log')

        super(FileToLogInterface, self).__init__(*args, **kwargs)

    def write(self, msg):
        super(FileToLogInterface, self).write(msg)
        self._logger.info(msg)
Rancor answered 30/1, 2015 at 12:48 Comment(0)
C
1

Programs written in other languages (e.g. C) have to do special magic (called double-forking) expressly to detach from the terminal (and to prevent zombie processes). So, I think the best solution is to emulate them.

A plus of re-executing your program is, you can choose redirections on the command-line, e.g. /usr/bin/python mycoolscript.py 2>&1 1>/dev/null

See this post for more info: What is the reason for performing a double fork when creating a daemon?

Chappell answered 8/6, 2013 at 17:23 Comment(1)
Eh... can't say I'm a fan of processes managing their own double-forking. It's so common an idiom, and so easy to code wrong if you aren't careful. Better to write your process to run in the foreground, and use a system background task manager (systemd, upstart) or other utility (daemon(1)) to handle the forking boilerplate.Hiawatha
Q
1

I know this question is answered (using python abc.py > output.log 2>&1 ), but I still have to say:

When writing your program, don't write to stdout. Always use logging to output whatever you want. That would give you a lot of freedom in the future when you want to redirect, filter, rotate the output files.

Quiver answered 16/7, 2021 at 13:47 Comment(0)
L
1

As mentioned by @jfs, most solutions will not properly handle some types of stdout output such as that from C extensions. There is a module that takes care of all this on PyPI called wurlitzer. You just need its sys_pipes context manager. It's as easy as using:

from contextlib import redirect_stdout
import os
from wurlitzer import sys_pipes
        
log = open("test.log", "a")
with redirect_stdout(log), sys_pipes():
    print("print statement")
    os.system("echo echo call")
Lacto answered 5/8, 2021 at 14:33 Comment(0)
R
0

Based on previous answers on this post I wrote this class for myself as a more compact and flexible way of redirecting the output of pieces of code - here just to a list - and ensure that the output is normalized afterwards.

class out_to_lt():
    def __init__(self, lt):
        if type(lt) == list:
            self.lt = lt
        else:
            raise Exception("Need to pass a list")            
    def __enter__(self):
        import sys
        self._sys = sys
        self._stdout = sys.stdout
        sys.stdout = self
        return self
    def write(self,txt):
        self.lt.append(txt)    
    def __exit__(self, type, value, traceback):
        self._sys.stdout = self._stdout

Used as:

lt = []
with out_to_lt(lt) as o:
    print("Test 123\n\n")
    print(help(str))

Updating. Just found a scenario where I had to add two extra methods, but was easy to adapt:

class out_to_lt():
    ...
    def isatty(self):
        return True #True: You're running in a real terminal, False:You're being piped, redirected, cron
    def flush(self):
        pass
Reyreyes answered 25/3, 2022 at 19:17 Comment(0)
O
0

There are other versions using context but nothing this simple. I actually just googled to double check it would work and was surprised not to see it, so for other people looking for a quick solution that is safe and directed at only the code within the context block, here it is:

import sys
with open('test_file', 'w') as sys.stdout:
    print('Testing 1 2 3')

Tested like so:

$ cat redirect_stdout.py
import sys

with open('test_file', 'w') as sys.stdout:
    print('Testing 1 2 3')
$ python redirect_stdout.py
$ cat test_file
Testing 1 2 3
Olivero answered 9/8, 2022 at 4:27 Comment(0)
P
0

For those interested, I extend the question. I need to write to log file for a while, then close it, rename it and use normal stdout afterwards. How do I do it?

print("Start program")
import os
import sys
sys.stdout.flush()
sys.stdout=open("xxxtmp", "wt")
print("xxx")
sys.stdout.close()
sys.stdout = sys.__stdout__
os.rename("xxxtmp", "xxx")
print("End program")

On stdout there will be:

Start program
End program

In xxx there will be:

xxx

If the program exits abnormally, the xxx file won't exist.

Prod answered 13/10, 2023 at 14:25 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.