Getting the remaining arguments in argparse

Question

I created an argparse.ArgumentParser and added specific arguments like:

parser.add_argument('-i', action='store', dest='i', default='i.log')
parser.add_argument('-o', action='store', dest='o', default='o.log')

But I want to be able to accept any arguments, not just the ones for which I explicitly specified the semantics.

How can I make it so that any remaining command-line arguments are gathered together and I can access them after parsing?

Cora

I don't quite understand what you're asking. Do you want all of the command line arguments not parsed by argparse? – Anatolic

I want to get all arguments, which i not take with using 'parser.add_arguments'. It's analog of default in switch – Cora

Answer 1

Another option is to add a positional argument to your parser. Specify the option without leading dashes, and argparse will look for them when no other option is recognized. This has the added benefit of improving the help text for the command:

>>> parser.add_argument('otherthings', nargs='*')
>>> parser.parse_args(['foo', 'bar', 'baz'])
Namespace(i='i.log', o='o.log', otherthings=['foo', 'bar', 'baz'])

and

>>> print parser.format_help()
usage: ipython-script.py [-h] [-i I] [-o O] [otherthings [otherthings ...]]

positional arguments:
  otherthings

optional arguments:
  -h, --help   show this help message and exit
  -i I
  -o O

Anatolic

Answer 2

Use parse_known_args():

args, unknownargs = parser.parse_known_args()

Askja

This is helpful when unknown arguments can have leading dashes. For example you are passing them on to another script. – Opt

Does not work if you want to allow passing conflicting argument names to a command you are wrapping, for example in docker-compose-wrapper.py --help vs docker-compose-wrapper.py exec --help – Bowden

There seem to be two solutions to extract possibly conflicting arguments for a wrapped command: 1) use argparse.REMAINDER to collect remaining arguments after the last known/expected, or 2) separate the wrapped command with -- on the command line. – Askja

This is useful when you want to forward args to something else – Copestone

Use argparse.REMAINDER when you want arguments to have an argument break requiring --. Use .parse_known_args when you want to transparently use multiple ArgumentParsers such as with a shared library that accepts arguments. – Schnapp

How can I get parser.print_help() adding a custom info in reasonable way about remaining unkown opts+args being forwarded to some other handler/program? – Cellulose

Answer 3

Use argparse.REMAINDER:

parser.add_argument('rest', nargs=argparse.REMAINDER)

Example:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-i', action='store', dest='i', default='i.log')
parser.add_argument('-o', action='store', dest='o', default='o.log')
parser.add_argument('rest', nargs=argparse.REMAINDER)
parser.parse_args(['hello', 'world'])
>>> Namespace(i='i.log', o='o.log', rest=['hello', 'world'])

Clamant

This is IMO the best solution for cases where you are wrapping another command: wrapper.py --help --> Your help text wrapper.py some args --help --> Passed on to the wrapped command – Bowden

This still raises an exception with optional (specified) args and a 'remaining' argument starting with '--'. The exception doesn't occur if you separate the extra args with --, but this is the same behaviour as using nargs="*". – Caladium

This works well, but the link in the answer (link) is broken in the sense that there's no dedicated argparse.REMAINDER section anymore. In fact, argparse.REMAINDER is only mentioned once on the entire page, in an unrelated way. Does this equate to argparse.REMAINDER being undocumented and, by now, unofficial? – Lianna

@AlexPovel You're right and it is kinda weird. It was there in 3.8, but not in the docs for 3.9. There is also no mention of it being deprecated in 3.9. I am running the latest python 3.9.2 and typing argparse.REMAINDER? in ipython shows that it is still there. It could just be a bug with the document generation process, or they are making some effort to eventually deprecate it. In any case, nothing points strongly to the feature no longer being part of python. – Clamant

argparse.REMAINDER is now undocumented and considered legacy but I don't believe it's going to be removed soon. See github.com/python/cpython/pull/18661 – Chimpanzee

If you are wrapping another command, also consider this answer about -- to pass all following arguments. You can use args = parser.parse_args() with rest = args.rest[1:] if args.rest[:1] == ["--"] else args.rest to remove the -- itself. – Symphonia

@Stavros Currently, it is not documented (v3.11.5). – Hassett

argparse.REMAINDER was last documented in Python 3.8: docs.python.org/3.8/library/argparse.html#argparse-remainder Newer versions of the docs don't include it. – Flocculant

Answer 4

I went and coded up the three suggestions given as answers in this thread. The test code appears at the bottom of this answer. Conclusion: The best all-around answer so far is nargs=REMAINDER, but it might really depend on your use case.

Here are the differences I observed:

(1) nargs=REMAINDER wins the user-friendliness test.

$ python test.py --help
Using nargs=*          : usage: test.py [-h] [-i I] [otherthings [otherthings ...]]
Using nargs=REMAINDER  : usage: test.py [-h] [-i I] ...
Using parse_known_args : usage: test.py [-h] [-i I]

(2) nargs=* quietly filters out the first -- argument on the command line, which seems bad. On the other hand, all methods respect -- as a way to say "please don't parse any more of these strings as known args."

$ ./test.py hello -- -- cruel -- -- world
Using nargs=*          : ['hello', '--', 'cruel', '--', '--', 'world']
Using nargs=REMAINDER  : ['hello', '--', '--', 'cruel', '--', '--', 'world']
Using parse_known_args : ['hello', '--', '--', 'cruel', '--', '--', 'world']

$ ./test.py -i foo -- -i bar
Using nargs=*          : ['-i', 'bar']
Using nargs=REMAINDER  : ['--', '-i', 'bar']
Using parse_known_args : ['--', '-i', 'bar']

(3) Any method except parse_known_args dies if it tries to parse a string beginning with - and it turns out not to be valid.

$ python test.py -c hello
Using nargs=*          : "unrecognized arguments: -c" and SystemExit
Using nargs=REMAINDER  : "unrecognized arguments: -c" and SystemExit
Using parse_known_args : ['-c', 'hello']

(4) nargs=REMAINDER completely stops parsing when it hits the first unknown argument. parse_known_args will gobble up arguments that are "known", no matter where they appear on the line (and will die if they look malformed).

$ python test.py hello -c world
Using nargs=*          : "unrecognized arguments: -c world" and SystemExit
Using nargs=REMAINDER  : ['hello', '-c', 'world']
Using parse_known_args : ['hello', '-c', 'world']

$ python test.py hello -i world
Using nargs=*          : ['hello']
Using nargs=REMAINDER  : ['hello', '-i', 'world']
Using parse_known_args : ['hello']

$ python test.py hello -i
Using nargs=*          : "error: argument -i: expected one argument" and SystemExit
Using nargs=REMAINDER  : ['hello', '-i']
Using parse_known_args : "error: argument -i: expected one argument" and SystemExit

Here's my test code.

#!/usr/bin/env python

import argparse
import sys

def using_asterisk(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument('-i', dest='i', default='i.log')
    parser.add_argument('otherthings', nargs='*')
    try:
        options = parser.parse_args(argv)
        return options.otherthings
    except BaseException as e:
        return e

def using_REMAINDER(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument('-i', dest='i', default='i.log')
    parser.add_argument('otherthings', nargs=argparse.REMAINDER)
    try:
        options = parser.parse_args(argv)
        return options.otherthings
    except BaseException as e:
        return e

def using_parse_known_args(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument('-i', dest='i', default='i.log')
    try:
        options, rest = parser.parse_known_args(argv)
        return rest
    except BaseException as e:
        return e

if __name__ == '__main__':
    print 'Using nargs=*          : %r' % using_asterisk(sys.argv[1:])
    print 'Using nargs=REMAINDER  : %r' % using_REMAINDER(sys.argv[1:])
    print 'Using parse_known_args : %r' % using_parse_known_args(sys.argv[1:])

Alysa

Another crucial difference is that argparse.ZERO_OR_MORE (*) will set the default passed to add_argument, but argparse.REMAINDER will not. It's also worth mentioning that argparse.REMAINDER is now an undocumented legacy feature due to "buggy" behavior. – Cochin

Answer 5

I've just stumbled across this problem. I solved it by removing the "--" arguments before argparse see's them.

# Extract underlying ("--") args before argparse parsing
for idx, arg in enumerate(argv):
    if arg == "--":
        wrapper_program_args = argv[:idx]
        underlying_tool_args = argv[idx + 1:]
        break
else:
    wrapper_program_args = argv
    underlying_tool_args = None

args = parser.parse_args(wrapper_program_args)
args.underlying_tool_args = underlying_tool_args

Optionally, if you want the help message to include the "--" option you may supply a custom formatter.

# Customer formatter required to print help message for "--" option.
class CustomHelpFormatter(argparse.HelpFormatter):
    def format_help(self):
        original_help = super().format_help()
        return original_help + "  --\t\t\tArguments to pass to underlying tool. \n"

parser = argparse.ArgumentParser(formatter_class=CustomHelpFormatter)

Marinamarinade