How to capture all arguments after double dash (--) using argparse

Question

Background

Many command-line utilities provide special handling for all arguments after a double dash (--). Examples:

git diff: All arguments after -- are paths (which can start with -):

git diff [options] -- [path...]

Some wrapper tools use this to pass arbitrary arguments on to the tool they are wrapping -- even arguments which may conflict with their own!

foowrap -a -- -a
        🠡     🠡
        │     └───  Passed to wrapped tool
        └─────────  Consumed by wrapper

Problem

How can we accomplish this using argparse?

Mistaken

Answer 1

It seems that argparse (specifically parse_known_args() already supports this natively, although it is not clearly documented:

import argparse

p = argparse.ArgumentParser()
p.add_argument("-a", action="store_true")
p.add_argument("-b", action="store_true")

args, extra = p.parse_known_args()
print(f"args: {args}")
print(f"extra: {extra}")

$ python3 known.py -a -- -a -b
args: Namespace(a=True, b=False)
extra: ['--', '-a', '-b']

We can see that:

• The -a after -- appears in extra
• The -b after -- appears in extra and does not set b=True in the namespace

If you want to ignore the -- separator, it can be filtered out:

extra = [a for a in extra if a != "--"]

---

_{I actually discovered this while attempting to code up a "what doesn't work" example. Because it is not obvious to me from the documentation I decided to ask+answer anyway.}

Mistaken

To ignore --, consider removing only the first instance, with something like if extra and extra[0] == '--': extra.pop(0); that way the user can still pass a literal "--" as an argument by doing python3 known.py -a -- --, for example so you can pass it along to a wrapped command. – Jarrettjarrid

Answer 2

I've just stumbled across this problem. I solved it by removing the "--" arguments before argparse see's them.

# Extract underlying ("--") args before argparse parsing
for idx, arg in enumerate(argv):
    if arg == "--":
        wrapper_program_args = argv[:idx]
        underlying_tool_args = argv[idx + 1:]
        break
else:
    wrapper_program_args = argv
    underlying_tool_args = None

args = parser.parse_args(wrapper_program_args)
args.underlying_tool_args = underlying_tool_args

Optionally, if you want the help message to include the "--" option you may supply a custom formatter.

# Customer formatter required to print help message for "--" option.
class CustomHelpFormatter(argparse.HelpFormatter):
    def format_help(self):
        original_help = super().format_help()
        return original_help + "  --\t\t\tArguments to pass to underlying tool. \n"

parser = argparse.ArgumentParser(formatter_class=CustomHelpFormatter)

U

Answer 3

You can add an argument with nargs=argparse.REMAINDER to capture the rest of the arguments.

parser = argparse.ArgumentParser()
# other arguments...
parser.add_argument('remaining', nargs=argparse.REMAINDER)
args = parser.parse_args()
# use args.remaining...

nargs='*' can also be used, which would not include the leading -- in the list.

Minelayer

argparse.REMAINDER is deprecated and no longer documented. It would be nice not to have to manually remove the leading -- though. – Mistaken

@JonathonReinhart You can use nargs='*' for that (as mentioned at the end of this answer). – Minelayer

My memory is that the handling of '--' and REMAINDER has changed with releases. I don't think REMAINDER is actually depricated, but it is now 'undocumentated' (i.e. most references removed from the docs). There were too many rough edges to its use. Looks like '--' behaves best with a * positional (last). – Impracticable

Both of our solutions suffer from the problem that a required positional argument after -- is still consumed and not put in remaining (rather than raising an error). – Mistaken