With Python's optparse module, how do you create an option that takes a variable number of arguments?

Question

With Perl's Getopt::Long you can easily define command-line options that take a variable number of arguments:

foo.pl --files a.txt             --verbose
foo.pl --files a.txt b.txt c.txt --verbose

Is there a way to do this directly with Python's optparse module? As far as I can tell, the nargs option attribute can be used to specify a fixed number of option arguments, and I have not seen other alternatives in the documentation.

Byplay

Specify filenames via arguments, not via option: foo.pl a.txt b.txt c.txt --verbose Filenames would be put in args in this case. – Hanlon

If --files defines inputs, this approach is not recommended. – Haunting

Answer 1

I believe optparse does not support what you require (not directly -- as you noticed, you can do it if you're willing to do all the extra work of a callback!-). You could also do it most simply with the third-party extension argparse, which does support variable numbers of arguments (and also adds several other handy bits of functionality).

This URL documents argparse's add_argument -- passing nargs='*' lets the option take zero or more arguments, '+' lets it take one or more arguments, etc.

Irita

Yep, but if you can't yet upgrade to 2.7 (released day before yesterday), the third-party package is still a godsend!-) – Irita

Answer 2

This took me a little while to figure out, but you can use the callback action to your options to get this done. Checkout how I grab an arbitrary number of args to the "--file" flag in this example.

from optparse import OptionParser,

def cb(option, opt_str, value, parser):
        args=[]
        for arg in parser.rargs:
                if arg[0] != "-":
                        args.append(arg)
                else:
                        del parser.rargs[:len(args)]
                        break
        if getattr(parser.values, option.dest):
                args.extend(getattr(parser.values, option.dest))
        setattr(parser.values, option.dest, args)

parser=OptionParser()
parser.add_option("-q", "--quiet",
        action="store_false", dest="verbose",
        help="be vewwy quiet (I'm hunting wabbits)")
parser.add_option("-f", "--filename",
        action="callback", callback=cb, dest="file")

(options, args) = parser.parse_args()

print options.file
print args

Only side effect is that you get your args in a list instead of tuple. But that could be easily fixed, for my particular use case a list is desirable.

Electrograph

I actually prefer the example in the python doc as shown in FMc answer: https://mcmap.net/q/890543/-with-python-39-s-optparse-module-how-do-you-create-an-option-that-takes-a-variable-number-of-arguments – Pitching

The two are functionally equivalent except the example code also allows values such as "-1" to be parsed as args to the flag which is a nice feature I suppose for some use cases. – Electrograph

indeed, it allows for negative numbers (I also like it comes straight from the official docs :) ) – Pitching

Answer 3

My mistake: just found this Callback Example 6.

Byplay

This is definitely the best answer for the exact question; argparse is much better, but you can't always easily use it: eg. in a Django management command. – Pitching

Answer 4

Wouldn't you be better off with this?

foo.pl --files a.txt,b.txt,c.txt --verbose

Anamorphism

No. Definitely not. For example, this would prohibit shell expansions like foo.py *.txt. – Oxidate

I don't think it deserves a -1, it could be valid solution in some cases, like the floating point list in documentation link above. Maybe the questioning tone is a bit off (but what do I know...). – Andreas

@Oxidate - Shell expansion is not always desirable. If you are dealing with large numbers of input files, you can hit an arbitrary command line length limit (made dynamic by the fact that it shrinks based on the size of your environment), in which case, it's better to use perl glob's bsd_glob method to do the expansion after you've received the args. I usually supply multiple files/glob patterns inside quotes, space delimited. – Winton

Answer 5

I recently has this issue myself: I was on Python 2.6 and needed an option to take a variable number of arguments. I tried to use Dave's solution but found that it wouldn't work without also explicitly setting nargs to 0.

def arg_list(option, opt_str, value, parser):
    args = set()
    for arg in parser.rargs:
        if arg[0] == '-':
            break
        args.add(arg)
        parser.rargs.pop(0)
    setattr(parser.values, option.dest, args)

parser=OptionParser()
parser.disable_interspersed_args()
parser.add_option("-f", "--filename", action="callback", callback=arg_list,
                  dest="file", nargs=0)

(options, args) = parser.parse_args()

The problem was that, by default, a new option added by add_options is assumed to have nargs = 1 and when nargs > 0 OptionParser will pop items off rargs and assign them to value before any callbacks are called. Thus, for options that do not specify nargs, rargs will always be off by one by the time your callback is called.

This callback is can be used for any number of options, just have callback_args be the function to be called instead of setattr.

Pilate

this one works for me with a small change : "if arg[0] == '-':" – Lyre

Novice reporting in... (stuck learning in legacy system) This returns every second option because of the .pop def arg_list(option, opt_str, value, parser): args = set() rargs = parser.rargs[:] for arg in rargs: etc... I tried to format it! – Finkle

Answer 6

Here's one way: Take the fileLst generating string in as a string and then use http://docs.python.org/2/library/glob.html to do the expansion ugh this might not work without escaping the *

Actually, got a better way: python myprog.py -V -l 1000 /home/dominic/radar/*.json <- If this is your command line

parser, opts = parse_args()

inFileLst = parser.largs

Oday