Consistent way to redirect both stdin & stdout to files in python using optparse

Question

I've got a dozen programs that can accept input via stdin or an option, and I'd like to implement the same features in a similar way for the output.

The optparse code looks like this:

parser.add_option('-f', '--file',
       default='-',
       help='Specifies the input file.  The default is stdin.')
parser.add_option('-o', '--output',
       default='-',
       help='Specifies the output file.  The default is stdout.')

The rest of the applicable code looks like this:

if opts.filename == '-':
    infile = sys.stdin
else:
    infile = open(opts.filename, "r")

if opts.output == '-':
    outfile = sys.stdout
else:
    outfile = open(opts.output, "w")

This code works fine and I like its simplicity - but I haven't been able to find a reference to anyone using a default value of '-' for output to indicate stdout. Is this a good consistent solution or am I overlooking something better or more expected?

Boogiewoogie

Small claritification - I've seen postings in the past in which a dash is recommended for an indicator in optparse/argparse that the program should read from stdin. But I haven't seen that for stdout. I've got no idea what the background is for this - sound familiar? Any insights? Otherwise, Adam Wagner's suggestion looks slightly more simple & readable. – Boogiewoogie

Answer 1

For input files you could use fileinput module. It follows common convention for input files: if no files given or filename is '-' it reads stdin, otherwise it reads from files given at a command-line.

There is no need in -f and --file options. If your program always requires an input file then it is not an option.

-o and --output is used to specify the output file name in various programs.

optparse

#!/usr/bin/env python
import fileinput
import sys
from optparse import OptionParser

parser = OptionParser()
parser.add_option('-o', '--output',
    help='Specifies the output file.  The default is stdout.')
options, files = parser.parse_args()
if options.output and options.output != '-':
   sys.stdout = open(options.output, 'w')

for line in fileinput.input(files):
    process(line)

argparse

argparse module allows you to specify explicitly files as arguments:

#!/usr/bin/env python
import fileinput
import sys
from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument('files', nargs='*', help='specify input files')
group = parser.add_mutually_exclusive_group()
group.add_argument('-o', '--output', 
    help='specify the output file.  The default is stdout')
group.add_argument('-i', '--inplace', action='store_true',
    help='modify files inplace')
args = parser.parse_args()

if args.output and args.output != '-':
   sys.stdout = open(args.output, 'w')

for line in fileinput.input(args.files, inplace=args.inplace):
    process(line)

Note: I've added --inplace option in the second example:

$ python util-argparse.py --help
usage: util-argparse.py [-h] [-o OUTPUT | -i] [files [files ...]]

positional arguments:
  files                 specify input files

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        specify the output file. The default is stdout
  -i, --inplace         modify files inplace

Schoolmaster

The fileinput & csv modules don't look compatible to me - so I'm not using fileinput for these programs. Am I wrong in this assumption? – Boogiewoogie

@KenFar: what do you mean by "compatible"? csv.reader(fileinput.input()) works as expected. – Schoolmaster

Whoops, my mistake. Looks like the best solution since this can also accept multiple input files. Thanks. – Boogiewoogie

@J.F. Sebastian , hi, I noticed that in the argpase example when the output is a file the code doesn't explicitly call close(), is this OK? – Joleenjolene

yes, you want to keep stdout open until your script ends. – Schoolmaster

Answer 2

If you can use argparse (i.e. Python 2.7+), it has built-in support for what you want: straight from argparse doc

The FileType factory creates objects that can be passed to the type argument of ArgumentParser.add_argument(). Arguments that have FileType objects as their type will open command-line arguments […] FileType objects understand the pseudo-argument '-' and automatically convert this into sys.stdin for readable FileType objects and sys.stdout for writable FileType objects.

So my advice is to simply use

import sys
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('file', type=argparse.FileType('r'),
    help="Specifies the input file")
parser.add_argument('output', type=argparse.FileType('w'),
    help="Specifies the output file")
args = parser.parse_args(sys.argv[1:])

# Here you can use your files
text = args.file.read()
args.output.write(text)
# … and so on

Then you can do

> python spam.py file output 

To read from file and output to output, or

> echo "Ni!" | python spam.py - output  

to read "Ni!" and output to output, or

> python spam.py file -

…

And it's good, since using - for the relevant stream is a convention that a lot of programs use. If you want to point it out, add it to the help strings.

  parser.add_argument('file', type=argparse.FileType('r'),
    help="Specifies the input file, '-' for standard input")

For reference, the usage message will be

> python spam.py -h
usage: [-h] file output

positional arguments:
  file        Specifies the input file, '-' for standard input
  output      Specifies the output file, '-' for standard output

optional arguments:
  -h, --help  show this help message and exit

Moldy