Python argparse positional arguments and sub-commands

Question

I'm working with argparse and am trying to mix sub-commands and positional arguments, and the following issue came up.

This code runs fine:

import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()

parser.add_argument('positional')
subparsers.add_parser('subpositional')

parser.parse_args('subpositional positional'.split())

The above code parses the args into Namespace(positional='positional'), however when I change the positional argument to have nargs='?' as such:

import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()

parser.add_argument('positional', nargs='?')
subparsers.add_parser('subpositional')

parser.parse_args('subpositional positional'.split())

It errors out with:

usage: [-h] {subpositional} ... [positional]
: error: unrecognized arguments: positional

Why is this?

Orvil

Btw, seems a known bug, that has been fixed for recent Python versions. – Nucleoside

Answer 1

At first I thought the same as jcollado, but then there's the fact that, if the subsequent (top level) positional arguments have a specific nargs (nargs = None, nargs = integer), then it works as you expect. It fails when nargs is '?' or '*', and sometimes when it is '+'. So, I went down to the code, to figure out what is going on.

It boils down to the way the arguments are split to be consumed. To figure out who gets what, the call to parse_args summarizes the arguments in a string like 'AA', in your case ('A' for positional arguments, 'O' for optional), and ends up producing a regex pattern to be matched with that summary string, depending on the actions you've added to the parser through the .add_argument and .add_subparsers methods.

In every case, for you example, the argument string ends up being 'AA'. What changes is the pattern to be matched (you can see the possible patterns under _get_nargs_pattern in argparse.py. For subpositional it ends up being '(-*A[-AO]*)', which means allow one argument followed by any number of options or arguments. For positional, it depends on the value passed to nargs:

• None => '(-*A-*)'
• 3 => '(-*A-*A-*A-*)' (one '-*A' per expected argument)
• '?' => '(-*A?-*)'
• '*' => '(-*[A-]*)'
• '+' => '(-*A[A-]*)'

Those patterns are appended and, for nargs=None (your working example), you end up with '(-*A[-AO]*)(-*A-*)', which matches two groups ['A', 'A']. This way, subpositional will parse only subpositional (what you wanted), while positional will match its action.

For nargs='?', though, you end up with '(-*A[-AO]*)(-*A?-*)'. The second group is comprised entirely of optional patterns, and * being greedy, that means the first group globs everything in the string, ending up recognizing the two groups ['AA', '']. This means subpositional gets two arguments, and ends up choking, of course.

Funny enough, the pattern for nargs='+' is '(-*A[-AO]*)(-*A[A-]*)', which works as long as you only pass one argument. Say subpositional a, as you require at least one positional argument in the second group. Again, as the first group is greedy, passing subpositional a b c d gets you ['AAAA', 'A'], which is not what you wanted.

In brief: a mess. I guess this should be considered a bug, but not sure what the impact would be if the patterns are turned into non-greedy ones...

Lusitania

Note that, of course, adding the subparsers after all the top level, as suggested by jcollado and argparses documentation, will break the ambiguity and work as intended! – Nucleoside

Answer 2

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional', nargs='?')

subparsers = parser.add_subparsers()
subparsers.add_parser('subpositional')

print(parser.parse_args(['positional', 'subpositional']))
# -> Namespace(positional='positional')
print(parser.parse_args(['subpositional']))
# -> Namespace(positional=None)
parser.print_usage()
# -> usage: bpython [-h] [positional] {subpositional} ...

The common practice is that arguments before the command (on the left side) belong to the main program, after (on the right) -- to the command. Therefore positional should go before the command subpositional. Example programs: git, twistd.

Additionally an argument with narg=? should probably be an option (--opt=value), and not a positional argument.

Terzas

What if the subparser had positional arguments? How can we let print(parser.parse_args(['subpositional', 'subparserarg'])) print: # -> Namespace(positional=None)? This should have said that we are selecting the subcommand 'subpositional' with the argument positional being optional. Is this possible? – Tew

Answer 3

I think that the problem is that when add_subparsers is called, a new parameter is added to the original parser to pass the name of the subparser.

For example, with this code:

import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()

parser.add_argument('positional')                                             
subparsers.add_parser('subpositional')                                             

parser.parse_args()

You get the following help string:

usage: test.py [-h] {subpositional} ... positional

positional arguments:
  {subpositional}
  positional

optional arguments:
  -h, --help       show this help message and exit

Note that subpositional is displayed before positional. I'd say that what you're looking for is to have the positional argument before the subparser name. Hence, probably what you're looking for is adding the argument before the subparsers:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')

subparsers = parser.add_subparsers()
subparsers.add_parser('subpositional')

parser.parse_args()

The help string obtained with this code is:

usage: test.py [-h] positional {subpositional} ...

positional arguments:
  positional
  {subpositional}

optional arguments:
  -h, --help       show this help message and exit

This way, you pass first the arguments to the main parser, then the name of the subparser and finally the arguments to the subparser (if any).

Pipette

It unfortunately doesn't seem to work. Help truly looks as it should, but in practice - it doesn't change the parsing process. – Polinski

Answer 4

It's still a mess in Python 3.5.

I suggest to subClass ArgumentParser to keep all the remaining positional arguments, and to deal with them later:

import argparse

class myArgumentParser(argparse.ArgumentParser):
    def parse_args(self, args=None, namespace=None):
       args, argv = self.parse_known_args(args, namespace)
       args.remaining_positionnals = argv
       return args

parser = myArgumentParser()

options = parser.parse_args()

The remaining positional arguments are in the list options.remaining_positionals

Jerky