How to handle both `with open(...)` and `sys.stdout` nicely?

Question

Often I need to output data either to file or, if file is not specified, to stdout. I use the following snippet:

if target:
    with open(target, 'w') as h:
        h.write(content)
else:
    sys.stdout.write(content)

I would like to rewrite it and handle both targets uniformly.

In ideal case it would be:

with open(target, 'w') as h:
    h.write(content)

but this will not work well because sys.stdout is be closed when leaving with block and I don't want that. I neither want to

stdout = open(target, 'w')
...

because I would need to remember to restore original stdout.

Related:

• Redirect stdout to a file in Python?
• Handling Exceptions - interesting article about handling exceptions in Python, as compared to C++

Edit

I know that I can wrap target, define separate function or use context manager. I look for a simple, elegant, idiomatic solution fitting that wouldn't require more than 5 lines

Pravit

Too bad you didn't add the edit earlier ;) Anyhow... alternatively you can simply not bother to cleanup your open file :P – Airt

Your first code snippet looks good to me: expresses intent and does what you want. – Symbolism

Consider cases where the file_like object h is used along several lines, not only one. Then the operations done to h should not be duplicated in the code! – Morsel

Answer 1

Just thinking outside of the box here, how about a custom open() method?

import sys
import contextlib

@contextlib.contextmanager
def smart_open(filename=None):
    if filename and filename != '-':
        fh = open(filename, 'w')
    else:
        fh = sys.stdout

    try:
        yield fh
    finally:
        if fh is not sys.stdout:
            fh.close()

Use it like this:

# For Python 2 you need this line
from __future__ import print_function

# writes to some_file
with smart_open('some_file') as fh:
    print('some output', file=fh)

# writes to stdout
with smart_open() as fh:
    print('some output', file=fh)

# writes to stdout
with smart_open('-') as fh:
    print('some output', file=fh)

Airt

This: https://mcmap.net/q/37750/-how-to-handle-both-with-open-and-sys-stdout-nicely is simpler. – Morsel

Answer 2

Stick with your current code. It's simple and you can tell exactly what it's doing just by glancing at it.

Another way would be with an inline if:

handle = open(target, 'w') if target else sys.stdout
handle.write(content)

if handle is not sys.stdout:
    handle.close()

But that isn't much shorter than what you have and it looks arguably worse.

You could also make sys.stdout unclosable, but that doesn't seem too Pythonic:

sys.stdout.close = lambda: None

with (open(target, 'w') if target else sys.stdout) as handle:
    handle.write(content)

Superorganic

You can keep unclosability for as long as you need it by making a context manager for it as well: with unclosable(sys.stdout): ... by setting sys.stdout.close = lambda: None inside this context manager and resetting it to the old value afterwards. But this seems a little bit too far fetched... – Predominant

I'm torn between voting up for "leave it, you can tell exactly what it's doing" and voting down for the horrendous unclosable suggestion! – Lulita

@Lulita I don't think that he was suggesting making sys.stdout unclosable, just noting tht it could be done. It's better to show bad ideas and explain why they're bad than not mention them and hope that they're not stumbled upon by others. – Dowzall

Answer 3

An improvement of Wolph's answer

import sys
import contextlib

@contextlib.contextmanager
def smart_open(filename: str, mode: str = 'r', *args, **kwargs):
    '''Open files and i/o streams transparently.'''
    if filename == '-':
        if 'r' in mode:
            stream = sys.stdin
        else:
            stream = sys.stdout
        if 'b' in mode:
            fh = stream.buffer  # type: IO
        else:
            fh = stream
        close = False
    else:
        fh = open(filename, mode, *args, **kwargs)
        close = True

    try:
        yield fh
    finally:
        if close:
            try:
                fh.close()
            except AttributeError:
                pass

This allows binary IO and pass eventual extraneous arguments to open if filename is indeed a file name.

Extroversion

Answer 4

As pointed in Conditional with statement in Python, Python 3.7 allows using contextlib.nullcontext for that:

from contextlib import nullcontext

with open(target, "w") if target else nullcontext(sys.stdout) as f:
    f.write(content)

Foreignborn

Answer 5

Why LBYL when you can EAFP?

try:
    with open(target, 'w') as h:
        h.write(content)
except TypeError:
    sys.stdout.write(content)

Why rewrite it to use the with/as block uniformly when you have to make it work in a convoluted way? You'll add more lines and reduce performance.

Snapp

Exceptions should not be used to control "normal" flow of the routine. Performance? will bubbling-up an error be faster that if/else? – Pravit

Depends on the probability that you'll be using one or the other. – Snapp

@JakubM. Exceptions can, should be and are used like this in Python. – Kerstinkerwin

@Lattyware: I added a link to interesting article about exceptions in Python. So you might be (partially, only partially!) right. Still, I wouldn't use exception here – Pravit

@JakubM.: Which part of that article conflicts with what Lattyware said? – Superorganic

None, it does not conflict. That's why I wrote he might be right – Pravit

Considering that Python's for loop exits by catching a StopIteration error thrown by the iterator it's looping across, I'd say that using exceptions for flow control is utterly Pythonic. – Dewain

Assuming that target is None when sys.stdout is intended, you need to catch TypeError rather than IOError. – Heald

@Heald Ah, bit of an oversight. (I don't think OP specified None but you'd want that I think, either that or he's slicing argv and if the last part isn't there he'd get an empty list.) – Snapp

obscure acronyms might be: Look Before You Leap and Easier to Ask Forgiveness than Permission. – Cribble

All my nature protests against the way of making an open() syscall before checking arguments in advance. – Andi

Indeed exceptions are often used this way in python, but I don't think that necessarily means it is a good idea for code readability. And there are a lot of ways you could end up with a TypeError, so if the try block had more complicated code in it, then this could result in output to both a file and to stdout. – Brake

Agreed that it is pythonic. Agreed that it is NOT the preferred pattern. What else might throw a TypeError? Is write() guaranteed to never throw a TypeError? For more complex programs you have a harder time guaranteeing that the catch block really does what you want it to. In simpler cases like KeyError, or custom exceptions like requests.ConnectionError you can reason much more easily about who might throw those exceptions, but TypeError et. al. are too broad IMHO. – Ministrant

Should I know what LBYL and EAFP mean? – Hyaline

Answer 6

Another possible solution: do not try to avoid the context manager exit method, just duplicate stdout.

with (os.fdopen(os.dup(sys.stdout.fileno()), 'w')
      if target == '-'
      else open(target, 'w')) as f:
      f.write("Foo")

Carberry

Answer 7

If it's fine that sys.stdout is closed after with body, you can also use patterns like this:

# Use stdout when target is "-"
with open(target, "w") if target != "-" else sys.stdout as f:
    f.write("hello world")

# Use stdout when target is falsy (None, empty string, ...)
with open(target, "w") if target else sys.stdout as f:
    f.write("hello world")

or even more generally:

with target if isinstance(target, io.IOBase) else open(target, "w") as f:
    f.write("hello world")

Bugbee

Answer 8

import contextlib
import sys

with contextlib.ExitStack() as stack:
    h = stack.enter_context(open(target, 'w')) if target else sys.stdout
    h.write(content)

Just two extra lines if you're using Python 3.3 or higher: one line for the extra import and one line for the stack.enter_context.

Lincolnlincolnshire

This is a nice, elegant solution that deserves more love. – Dodge

Answer 9

I'd also go for a simple wrapper function, which can be pretty simple if you can ignore the mode (and consequently stdin vs. stdout), for example:

from contextlib import contextmanager
import sys

@contextmanager
def open_or_stdout(filename):
    if filename != '-':
        with open(filename, 'w') as f:
            yield f
    else:
        yield sys.stdout

Readership

This solution doesn't explicitly close the file either on normal or error termination of the with clause so its not much of a context manager. A class that implements enter and exit would be a better choice. – Bidwell

I get ValueError: I/O operation on closed file if I try to write to the file outside the with open_or_stdout(..) block. What am I missing? sys.stdout is not meant to be closed. – Readership

Answer 10

Okay, if we are getting into one-liner wars, here's:

(target and open(target, 'w') or sys.stdout).write(content)

I like Jacob's original example as long as context is only written in one place. It would be a problem if you end up re-opening the file for many writes. I think I would just make the decision once at the top of the script and let the system close the file on exit:

output = target and open(target, 'w') or sys.stdout
...
output.write('thing one\n')
...
output.write('thing two\n')

You could include your own exit handler if you think its more tidy

import atexit

def cleanup_output():
    global output
    if output is not sys.stdout:
        output.close()

atexit(cleanup_output)

Bidwell

I don't think your one-liner closes the file object. Am I wrong? – Snapp

@Snapp - It does... conditionally. The file object's refcount goes to zero because there are no variables pointing to it, so it is available to have its __del__ method called either immediately (in cpython) or later when garbage collection happens. There are warnings in the doc not to trust that this will always work but I use it all the time in shorter scripts. Something big that runs a long time and opens lots of files... well I guess I'd use 'with' or 'try/finally'. – Bidwell

TIL. I didn't know that file objects' __del__ would do that. – Snapp

@2rs2ts: CPython uses a reference-counting garbage collector (with a "real" GC underneath invoked as needed) so it can close the file as soon as you drop all references to the stream-handle. Jython and apparently IronPython only have the "real" GC so they don't close the file until an eventual GC. – Heald

Answer 11

This is a simpler and shorter version of the accepted answer

import contextlib, sys


def writer(fn): 
    @contextlib.contextmanager
    def stdout():
        yield sys.stdout
    return open(fn, 'w') if fn else stdout()

usage:

with writer('') as w:
    w.write('hello\n')

with writer('file.txt') as w:
    w.write('hello\n')

Morsel

Answer 12

If you really must insist on something more "elegant", i.e. a one-liner:

>>> import sys
>>> target = "foo.txt"
>>> content = "foo"
>>> (lambda target, content: (lambda target, content: filter(lambda h: not h.write(content), (target,))[0].close())(open(target, 'w'), content) if target else sys.stdout.write(content))(target, content)

foo.txt appears and contains the text foo.

Snapp

This should be moved to CodeGolf StackExchange :D – Vallonia

Answer 13

How about opening a new fd for sys.stdout? This way you won't have any problems closing it:

if not target:
    target = "/dev/stdout"
with open(target, 'w') as f:
    f.write(content)

Idoux

Sadly, running this python script needs a sudo on my install. /dev/stdout is owned by root. – Nathalia

In many situations, re-opening an fd to stdout is not what's expected. For example, this code will truncate stdout, thus making shell things like ./script.py >> file overwrite the file instead of appending to it. – Tarter

This won't work on windows which has no /dev/stdout. – Heterosexual

Answer 14

if (out != sys.stdout):
    with open(out, 'wb') as f:
        f.write(data)
else:
    out.write(data)

Slight improvement in some cases.

Charlyncharm

Answer 15

The following solution is not a beauty, but from a time long, long ago; just before with ...

handler = open(path, mode = 'a') if path else sys.stdout
try:
    print('stuff', file = handler)
    ... # other stuff or more writes/prints, etc.
except Exception as e:
    if not (path is None): handler.close()
    raise e
handler.close()

Doeskin

Answer 16

One way to solve it is with polymorphism. Pathlib.path has an open method that functions as you would expect:

from pathlib import Path

output = Path("/path/to/file.csv")

with output.open(mode="w", encoding="utf-8") as f:
    print("hello world", file=f)

we can copy this interface for printing

import sys

class Stdout:
    def __init__(self, *args):
        pass

    def open(self, mode=None, encoding=None):
        return self

    def __enter__(self):
        return sys.stdout

    def __exit__(self, exc_type, exc_value, traceback):
        pass

Now we simply replace Path with Stdout

output = Stdout("/path/to/file.csv")

with output.open(mode="w", encoding="utf-8") as f:
    print("hello world", file=f)

This isn't necessarily better than overloading open, but it's a convenient solution if you're using Path objects.

Ethban

Answer 17

With python 3 you can used wrap stdout file descriptor with IO object and avoid closing on context leave it with closefd=False:

h = open(target, 'w') if target else open(sys.stdout.fileno(), 'w', closefd=False)

with h as h:
    h.write(content)

Directorate