Multiple invocation of the same subcommand in a single command line

Question

I'm trying to figure out how to use argparser to do the following:

$ python test.py executeBuild --name foobar1 executeBuild --name foobar2 ....

getBuild itself is a sub-command. My goal is to have the script have the capability to chain a series of sub-command (executeBuild being one of them) and execute them in order. In the example above, it would execute a build, then setup the environment, then execute build again. How can I accomplish this with argparse? I've tried the following:

    main_parser = argparse.ArgumentParser(description='main commands')
    subparsers = main_parser.add_subparsers(help='SubCommands', dest='command')

    build_parser = subparsers.add_parser('executeBuild')
    build_parser.add_argument('--name', action='store', nargs=1, dest='build_name')
    check_parser = subparsers.add_parser('setupEnv')

    args, extra=main_parser.parse_known_args() 

However, it appears that whenever I do this, it goes into the subcommand of executeBuild and report it doesn't know what executeBuild is. I've tried parsing out the extra so I can do a repeat call / chain, however, the first view property appears to have been overwritten, so I can't even just save the extra options and iterate thru.

Sofer

The 2nd 'executeBuild' appears in extra because it can't be parsed (i.e. build_parser does not accept a positional). But the 2nd '--name' is recognized by the build_parser, and treated just like the 1st. In other words, a parser puts things it can't recognized in 'extra' but keeps going. 'optionals' can be in any order and can be repeated. – Uncanny

Answer 1

You are asking argparse something it was not written for : it is good at parsing one command line (but only one) and you want to parse multiple commands in one single line. IMHO, you have to do an initial splitting on your arguments array, and then use argparse on each subcommand. Following function takes a list of arguments (could be sys.argv), skips the first and split remaining in arrays beginning on each known subcommand. You can then use argparse on each sublist :

def parse(args, subcommands):
    cmds = []
    cmd = None
    for arg in args[1:]:
        if arg in (subcommands):
            if cmd is not None:
                cmds.append(cmd)
            cmd = [arg]
        else:
            cmd.append(arg)
    cmds.append(cmd)
    return cmds

In your example :

parse(['test.py', 'executeBuild', '--name', 'foobar1', 'executeBuild', '--name', 'foobar2'],
    ('executeBuild',))

=>

[['executeBuild', '--name', 'foobar1'], ['executeBuild', '--name', 'foobar2']]

Limits : subcommands are used as reserved words and cannot be used as option arguments.

Made

Answer 2

Splitting sys.argv before hand is a good solution. But it can also be done while parsing using an argument with nargs=argparse.REMAINDER. This type of argument gets the rest of the strings, regardless of whether they look like flags or not.

Replacing the parse_known_args with this code:

...
build_parser.add_argument('rest', nargs=argparse.REMAINDER)
check_parser.add_argument('rest', nargs=argparse.REMAINDER)
extras = 'executeBuild --name foobar1 setupEnv executeBuild --name foobar2'.split()
# or extras = sys.argv[1:]
while extras:
    args = main_parser.parse_args(extras)
    extras = args.rest
    delattr(args,'rest')
    print args
    # collect args as needed

prints:

Namespace(build_name=['foobar1'], command='executeBuild')
Namespace(command='setupEnv')
Namespace(build_name=['foobar2'], command='executeBuild')

In the documentation:

argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:

A problem with REMAINDER is that can be too greedy. http://bugs.python.org/issue14174. As a result build_parser and check_parser can't have other positional arguments.

---

A way around the greedy REMAINDER is to use argparse.PARSER. This is the nargs value that subparsers uses (undocumented). It's like REMAINDER, except that the first string must look like an 'argument' (no '-'), and is matched against choices (if given). PARSER isn't as greedy as REMAINDER, so the subparsers can have other positional arguments.

There's some extra code involving an 'exit' string and dummy parser. This is to get around the fact that the PARSER argument is 'required' (somewhat like nargs='+')

from argparse import ArgumentParser, PARSER, SUPPRESS

main_parser = ArgumentParser(prog='MAIN')
parsers = {'exit': None}
main_parser.add_argument('rest',nargs=PARSER, choices=parsers)

build_parser = ArgumentParser(prog='BUILD')
parsers['executeBuild'] = build_parser
build_parser.add_argument('cmd')
build_parser.add_argument('--name', action='store', nargs=1, dest='build_name')
build_parser.add_argument('rest',nargs=PARSER, choices=parsers, help=SUPPRESS)

check_parser = ArgumentParser(prog='CHECK')
parsers['setupEnv'] = check_parser
check_parser.add_argument('cmd')
check_parser.add_argument('foo')
check_parser.add_argument('rest',nargs=PARSER, choices=parsers, help=SUPPRESS)

argv = sys.argv[1:]
if len(argv)==0:
    argv = 'executeBuild --name foobar1 setupEnv foo executeBuild --name foobar2'.split()
argv.append('exit') # extra string to properly exit the loop
parser = main_parser
while parser:
    args = parser.parse_args(argv)
    argv = args.rest
    delattr(args,'rest')
    print(parser.prog, args)
    parser = parsers.get(argv[0], None)

sample output:

('MAIN', Namespace())
('BUILD', Namespace(build_name=['foobar1'], cmd='executeBuild'))
('CHECK', Namespace(cmd='setupEnv', foo='foo'))
('BUILD', Namespace(build_name=['foobar2'], cmd='executeBuild'))

---

Another possibility is to use '--' to separate command blocks:

'executeBuild --name foobar1 -- setupEnv -- executeBuild --name foobar2'

However there is problem when there are several '--': http://bugs.python.org/issue13922

Uncanny