Can I redirect the stdout into some sort of string buffer?

Question

I'm using python's ftplib to write a small FTP client, but some of the functions in the package don't return string output, but print to stdout. I want to redirect stdout to an object which I'll be able to read the output from.

I know stdout can be redirected into any regular file with:

stdout = open("file", "a")

But I prefer a method that doesn't uses the local drive.

I'm looking for something like the BufferedReader in Java that can be used to wrap a buffer into a stream.

Monovalent

I do not think stdout = open("file", "a") by itself will redirect anything. – Franko

Answer 1

from cStringIO import StringIO # Python3 use: from io import StringIO
import sys

old_stdout = sys.stdout
sys.stdout = mystdout = StringIO()

# blah blah lots of code ...

sys.stdout = old_stdout

# examine mystdout.getvalue()

Emia

+1, you don't need to keep a reference to the original stdout object, as it is always available at sys.__stdout__. See docs.python.org/library/sys.html#sys.__stdout__. – Actress

Well, that's an interesting debate. The absolute original stdout is available, but when replacing like this, it's better to use an explicit save as I've done, since someone else could have replaced stdout and if you use stdout, you'd clobber their replacement. – Emia

would this operation in one thread alter the behavior of other threads? I mean is it threadsafe? – Amoral

@AnuvratParashar: I think that would be an excellent question to ask on its own. – Eke

@AnuvratParashar: I'm pretty sure it is not thread safe, unless there are details I don't know about (or didn't notice), I used this method to retrieve print calls in other threads. – Bronnie

This doesn't work if sys.stdout.buffer (Python 3) is used. See my answer for a solution working in that case. For old Python-2-only code this is still the best solution. – Wollis

I highly recommend to reassign the old stdout in a finally: block, so it is also reassigned if an exception is risen in between. try: bkp = sys.stdout ... ... finally: sys.stdout = bkp – Piccalilli

@erikb85: subprocess.call() requires that sys.stdout.fileno() is redirected, the solution that just replaces sys.stdout won't work even for functions that store sys.stdout value locally in Python. See stdout_redirected() that redirects sys.stdout.fileno() instead of replacing sys.stdout. Though if it is your code, you could use subprocess.call(stdout=..) to redirect stdout of a subprocess (or just call subprocess.check_output()). – Remission

+1 for @NedBatchelder comment to store stdout. E.g. in Jupyter notebooks, stdout is altered and resetting to sys.__stdout__ screws up printing in Jupyter. – Azzieb

Replacing built-in library attributes is bad practice (a.o. because of thread safety) – Concordance

Answer 2

There is a contextlib.redirect_stdout() function in Python 3.4+:

import io
from contextlib import redirect_stdout

with io.StringIO() as buf, redirect_stdout(buf):
    print('redirected')
    output = buf.getvalue()

Here's a code example that shows how to implement it on older Python versions.

Remission

There's also redirect_stderr on the latest Python too! – Seamount

Answer 3

Just to add to Ned's answer above: you can use this to redirect output to any object that implements a write(str) method.

This can be used to good effect to "catch" stdout output in a GUI application.

Here's a silly example in PyQt:

import sys
from PyQt4 import QtGui

class OutputWindow(QtGui.QPlainTextEdit):
    def write(self, txt):
        self.appendPlainText(str(txt))

app = QtGui.QApplication(sys.argv)
out = OutputWindow()
sys.stdout=out
out.show()
print "hello world !"

Encephalograph

Works for me with python 2.6 and PyQT4. Seems strange to down vote working code when you can't tell why it doesn't work ! – Encephalograph

don't forget to add flush() too! – Garlaand

Answer 4

A context manager for python3:

import sys
from io import StringIO


class RedirectedStdout:
    def __init__(self):
        self._stdout = None
        self._string_io = None

    def __enter__(self):
        self._stdout = sys.stdout
        sys.stdout = self._string_io = StringIO()
        return self

    def __exit__(self, type, value, traceback):
        sys.stdout = self._stdout

    def __str__(self):
        return self._string_io.getvalue()

use like this:

>>> with RedirectedStdout() as out:
>>>     print('asdf')
>>>     s = str(out)
>>>     print('bsdf')
>>> print(s, out)
'asdf\n' 'asdf\nbsdf\n'

Fronde

Answer 5

Starting with Python 2.6 you can use anything implementing the io.TextIOBase API as a replacement. This solution also enables you to use sys.stdout.buffer.write() in Python 3 to write (already) encoded byte strings to standard output (see stdout in Python 3). Using io.StringIO wouldn't work then, because neither sys.stdout.buffer nor sys.stdout.encoding would be available.

A solution using io.TextIOWrapper:

import io
import sys

# Setup stdout.
old_stdout = sys.stdout
sys.stdout = io.TextIOWrapper(io.BytesIO(), sys.stdout.encoding)

# Write to stdout or stdout.buffer.
...

# Read from stdout.
sys.stdout.seek(0)
out = sys.stdout.read()

# Restore stdout.
sys.stdout.close()
sys.stdout = old_stdout

This solution works for Python 2 >= 2.6 and Python 3.

Please note that the new sys.stdout.write() only accepts unicode strings and sys.stdout.buffer.write() only accepts byte strings. Code that is built to run on Python 2 and 3 without changes often makes use of sys.stdout.buffer.

So to have sys.stdout.write() accept both unicode and byte strings, you can instead use this io.TextIOWrapper subclass:

class StdoutBuffer(io.TextIOWrapper):

    def write(self, string):
        try:
            return super(StdoutBuffer, self).write(string)
        except TypeError:
            # Redirect encoded byte strings directly to buffer.
            return super(StdoutBuffer, self).buffer.write(string)

You don't have to set the encoding of the buffer the sys.stdout.encoding, but this helps when using this method for testing/comparing script output.

Wollis

This answer helped me when setting up an Environment object's stdout param for use with Httpie's core.py. – Taphouse

Answer 6

This method restores sys.stdout even if there's an exception. It also gets any output before the exception.

import io
import sys

real_stdout = sys.stdout
fake_stdout = io.BytesIO()   # or perhaps io.StringIO()
try:
    sys.stdout = fake_stdout
    # do what you have to do to create some output
finally:
    sys.stdout = real_stdout
    output_string = fake_stdout.getvalue()
    fake_stdout.close()
    # do what you want with the output_string

Tested in Python 2.7.10 using io.BytesIO()

Tested in Python 3.6.4 using io.StringIO()

---

Bob, added for a case if you feel anything from the modified / extended code experimentation might get interesting in any sense, otherwise feel free to delete it

^{Ad informandum ... a few remarks from extended experimentation during finding some viable mechanics to "grab" outputs, directed by numexpr.print_versions() directly to the <stdout> ( upon a need to clean GUI and collecting details into debugging-report )}

# THIS WORKS AS HELL: as Bob Stein proposed years ago:
#  py2 SURPRISEDaBIT:
#
import io
import sys
#
real_stdout = sys.stdout                        #           PUSH <stdout> ( store to REAL_ )
fake_stdout = io.BytesIO()                      #           .DEF FAKE_
try:                                            # FUSED .TRY:
    sys.stdout.flush()                          #           .flush() before
    sys.stdout = fake_stdout                    #           .SET <stdout> to use FAKE_
    # ----------------------------------------- #           +    do what you gotta do to create some output
    print 123456789                             #           + 
    import  numexpr                             #           + 
    QuantFX.numexpr.__version__                 #           + [3] via fake_stdout re-assignment, as was bufferred + "late" deferred .get_value()-read into print, to finally reach -> real_stdout
    QuantFX.numexpr.print_versions()            #           + [4] via fake_stdout re-assignment, as was bufferred + "late" deferred .get_value()-read into print, to finally reach -> real_stdout
    _ = os.system( 'echo os.system() redir-ed' )#           + [1] via real_stdout                                 + "late" deferred .get_value()-read into print, to finally reach -> real_stdout, if not ( _ = )-caught from RET-d "byteswritten" / avoided from being injected int fake_stdout
    _ = os.write(  sys.stderr.fileno(),         #           + [2] via      stderr                                 + "late" deferred .get_value()-read into print, to finally reach -> real_stdout, if not ( _ = )-caught from RET-d "byteswritten" / avoided from being injected int fake_stdout
                       b'os.write()  redir-ed' )#  *OTHERWISE, if via fake_stdout, EXC <_io.BytesIO object at 0x02C0BB10> Traceback (most recent call last):
    # ----------------------------------------- #           ?                              io.UnsupportedOperation: fileno
    #'''                                                    ? YET:        <_io.BytesIO object at 0x02C0BB10> has a .fileno() method listed
    #>>> 'fileno' in dir( sys.stdout )       -> True        ? HAS IT ADVERTISED,
    #>>> pass;            sys.stdout.fileno  -> <built-in method fileno of _io.BytesIO object at 0x02C0BB10>
    #>>> pass;            sys.stdout.fileno()-> Traceback (most recent call last):
    #                                             File "<stdin>", line 1, in <module>
    #                                           io.UnsupportedOperation: fileno
    #                                                       ? BUT REFUSES TO USE IT
    #'''
finally:                                        # == FINALLY:
    sys.stdout.flush()                          #           .flush() before ret'd back REAL_
    sys.stdout = real_stdout                    #           .SET <stdout> to use POP'd REAL_
    sys.stdout.flush()                          #           .flush() after  ret'd back REAL_
    out_string = fake_stdout.getvalue()         #           .GET string           from FAKE_
    fake_stdout.close()                         #                <FD>.close()
    # +++++++++++++++++++++++++++++++++++++     # do what you want with the out_string
    #
    print "\n{0:}\n{1:}{0:}".format( 60 * "/\\",# "LATE" deferred print the out_string at the very end reached -> real_stdout
                                     out_string #                   
                                     )
'''
PASS'd:::::
...
os.system() redir-ed
os.write()  redir-ed
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
123456789
'2.5'
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Numexpr version:   2.5
NumPy version:     1.10.4
Python version:    2.7.13 |Anaconda 4.0.0 (32-bit)| (default, May 11 2017, 14:07:41) [MSC v.1500 32 bit (Intel)]
AMD/Intel CPU?     True
VML available?     True
VML/MKL version:   Intel(R) Math Kernel Library Version 11.3.1 Product Build 20151021 for 32-bit applications
Number of threads used by default: 4 (out of 4 detected cores)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
>>>

EXC'd :::::
...
os.system() redir-ed
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
123456789
'2.5'
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Numexpr version:   2.5
NumPy version:     1.10.4
Python version:    2.7.13 |Anaconda 4.0.0 (32-bit)| (default, May 11 2017, 14:07:41) [MSC v.1500 32 bit (Intel)]
AMD/Intel CPU?     True
VML available?     True
VML/MKL version:   Intel(R) Math Kernel Library Version 11.3.1 Product Build 20151021 for 32-bit applications
Number of threads used by default: 4 (out of 4 detected cores)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
io.UnsupportedOperation: fileno
'''

Guzzle

Answer 7

In Python3.6, the StringIO and cStringIO modules are gone, you should use io.StringIO instead.So you should do this like the first answer:

import sys
from io import StringIO

old_stdout = sys.stdout
old_stderr = sys.stderr
my_stdout = sys.stdout = StringIO()
my_stderr = sys.stderr = StringIO()

# blah blah lots of code ...

sys.stdout = self.old_stdout
sys.stderr = self.old_stderr

// if you want to see the value of redirect output, be sure the std output is turn back
print(my_stdout.getvalue())
print(my_stderr.getvalue())

my_stdout.close()
my_stderr.close()

Discrepancy

You could improve the quality of your Answer by explaining how the above code works and how this is an improvement over the Questioner's situation. – Astrology

Answer 8

Use pipe() and write to the appropriate file descriptor.

https://docs.python.org/library/os.html#file-descriptor-operations

Dropout

Answer 9

Here's another take on this. contextlib.redirect_stdout with io.StringIO() as documented is great, but it's still a bit verbose for every day use. Here's how to make it a one-liner by subclassing contextlib.redirect_stdout:

import sys
import io
from contextlib import redirect_stdout

class capture(redirect_stdout):

    def __init__(self):
        self.f = io.StringIO()
        self._new_target = self.f
        self._old_targets = []  # verbatim from parent class

    def __enter__(self):
        self._old_targets.append(getattr(sys, self._stream))  # verbatim from parent class
        setattr(sys, self._stream, self._new_target)  # verbatim from parent class
        return self  # instead of self._new_target in the parent class

    def __repr__(self):
        return self.f.getvalue()  

Since __enter__ returns self, you have the context manager object available after the with block exits. Moreover, thanks to the __repr__ method, the string representation of the context manager object is, in fact, stdout. So now you have,

with capture() as message:
    print('Hello World!')
print(str(message)=='Hello World!\n')  # returns True

Mclellan