Python: argument parser that handles global options to sub-commands properly

Question

argparse fails at dealing with sub-commands receiving global options:

import argparse
p = argparse.ArgumentParser()
p.add_argument('--arg', action='store_true')
s = p.add_subparsers()
s.add_parser('test')

will have p.parse_args('--arg test'.split()) work,
but fails on p.parse_args('test --arg'.split()).

Anyone aware of a python argument parser that handles global options to sub-commands properly?

Copyreader

What do you mean by 'fails'? What do you want to happen? What global options? – Pulmonary

fail as in complaining about the unknown argument --arg – Copyreader

Answer 1

Give docopt a try:

>>> from docopt import docopt

>>> usage = """
... usage: prog.py command [--test]
...        prog.py another [--test]
... 
... --test  Perform the test."""

>>> docopt(usage, argv='command --test')
{'--test': True,
 'another': False,
 'command': True}

>>> docopt(usage, argv='--test command')
{'--test': True,
 'another': False,
 'command': True}

Brebner

@Roony, great! and thanks for you feedback, more of that is always welcome. – Brebner

Answer 2

You can easily add this argument to both parsers (main parser and subcommand parser):

import argparse                                                                  

main = argparse.ArgumentParser()                                                    
subparser = main.add_subparsers().add_parser('test')                                        

for p in [main,subparser]:                                                                  
   p.add_argument('--arg', action='store_true')                                 

print main.parse_args('--arg test'.split()).arg                                     
print main.parse_args('test --arg'.split()).arg

Edit: As @hpaulj pointed in comment, there is also parents argument which you can pass to ArgumentParser constructor or to add_parser method. You can list in this value parsers which are bases for new one.

import argparse

base = argparse.ArgumentParser(add_help=False)
base.add_argument('--arg', action='store_true')

main = argparse.ArgumentParser(parents=[base])
subparser = main.add_subparsers().add_parser('test', parents=[base])

print main.parse_args('--arg test'.split()).arg
print main.parse_args('test --arg'.split()).arg

More examples/docs:

looking for best way of giving command line arguments in python, where some params are req for some option and some params are req for other options

Python argparse - Add argument to multiple subparsers (I'm not sure if this question is not overlaping with this one too much)

http://docs.python.org/dev/library/argparse.html#parents

Santos

If there were more optionals, they could be defined in a parent parser, and added to both p and s via the parents parameter. https://mcmap.net/q/1172910/-looking-for-best-way-of-giving-command-line-arguments-in-python-where-some-params-are-req-for-some-option-and-some-params-are-req-for-other-options has an example of adding the same set of optionals to many subparsers. – Genous

@Genous - I think that this is even better answer! I'm going to update my response... – Santos

On Python 2.7.16, the second example prints False in the first case, which is, I believe, not at all the desired behavior? Therefore I think the second, edited solution is not correct. – Glide

Answer 3

There's a ton of argument-parsing libs in the Python world. Here are a few that I've seen, all of which should be able to handle address the problem you're trying to solve (based on my fuzzy recollection of them when I played with them last):

• opster—I think this is what mercurial uses, IIRC
• docopt—This one is new, but uses an interesting approach
• cliff—This is a relatively new project by Doug Hellmann (PSF member, virtualenvwrapper author, general hacker extraordinaire) is a bit more than just an argument parser, but is designed from the ground up to handle multi-level commands
• clint—Another project that aims to be "argument parsing and more", this one by Kenneth Reitz (of Requests fame).

Hispanicism

Wow -- I didn't realize there were so many options -- I guess that's probably because I've always been able to coax argparse into submission. +1 for finding all these though... – Dronski

i did a quick investigation, docopt and clint wont do cliff is a very strange and complex thing i dont want to have to understand opster looks interesting, trying it – Copyreader

@Ronny, docopt should work—you can add [options] shortcut, it allows to pass global options regardless of sub-command hierarchy. – Brebner

@Ronny i.e. case 'test --arg' will be recognized in case you specify pattern my_program test [options] or my_program test [--arg]. – Brebner

Answer 4

Here's a dirty workaround --

import argparse
p = argparse.ArgumentParser()
p.add_argument('--arg', action='store_true')
s = p.add_subparsers()
s.add_parser('test')

def my_parse_args(ss):
    #parse the info the subparser knows about; don't issue an error on unknown stuff
    namespace,leftover=p.parse_known_args(ss) 
    #reparse the unknown as global options and add it to the namespace.
    if(leftover):
        s.add_parser('null',add_help=False)
        p.parse_args(leftover+['null'],namespace=namespace)

    return namespace

#print my_parse_args('-h'.split())  #This works too, but causes the script to stop.
print my_parse_args('--arg test'.split())
print my_parse_args('test --arg'.split())

This works -- And you could modify it pretty easily to work with sys.argv (just remove the split string "ss"). You could even subclass argparse.ArgumentParser and replace the parse_args method with my_parse_args and then you'd never know the difference -- Although subclassing to replace a single method seems overkill to me.

I think however, that this is a lit bit of a non-standard way to use subparsers. In general, global options are expected to come before subparser options, not after.

Dronski

Answer 5

The parser has a specific syntax: command <global options> subcommand <subcommand ptions>, you are trying to feed the subcommand with an option and but you didn't define one.

List

I think Ronny is aware of why it fails (or at least he doesn't care) -- he's looking for a workaround (using argparse or something else). – Dronski