Can't get argparse to read quoted string with dashes in it?

Question

Is there a way to make argparse recognize anything between two quotes as a single argument? It seems to keep seeing the dashes and assuming that it's the start of a new option

I have something like:

mainparser = argparse.ArgumentParser()
subparsers = mainparser.add_subparsers(dest='subcommand')
parser = subparsers.add_parser('queue')
parser.add_argument('-env', '--extraEnvVars', type=str,
                        help='String of extra arguments to be passed to model.')
...other arguments added to parser...

But when I run:

python Application.py queue -env "-s WHATEVER -e COOL STUFF"

It gives me:

Application.py queue: error: argument -env/--extraEnvVars: expected one argument

If I leave off the first dash, it works totally fine, but it's kind of crucial that I be able to pass in a string with dashes in it. I've tried escaping it with \ , which causes it to succeed but adds the \ to the argument string Does anyone know how to get around this? This happens whether or not -s is an argument in parser.

EDIT: I'm using Python 2.7.

EDIT2:

python Application.py -env " -env"

works perfectly fine, but

python Application.py -env "-env"

does not.

EDIT3: Looks like this is actually a bug that's being debated already: http://www.gossamer-threads.com/lists/python/bugs/89529, http://python.6.x6.nabble.com/issue9334-argparse-does-not-accept-options-taking-arguments-beginning-with-dash-regression-from-optp-td578790.html. It's only in 2.7 and not in optparse.

EDIT4: The current open bug report is: http://bugs.python.org/issue9334

Shanika

What version of Python are you using? – Unpeople

I'm using Python 2.7. – Shanika

This works fine for me on Python 2.7. Do you have any other arguments configured? – Annoying

Yes, a number of them. Also, -e is the argument of one of the subparsers of my program. I'll post a more complete code snippet to make it clearer. – Shanika

Hm...pretty sure. All of my other options work fine, and extraEnvVars does what it's supposed to do as long as the quoted string doesn't start with a dash. For example, python Application.py queue -env " -env" works fine. – Shanika

BTW, "quoted" here is entirely moot -- the quotes are consumed by the shell before argparse (or the Python interpreter) has any chance to well whether they're there or not. – Drolet

Answer 1

You can start the argument with a space python tst.py -e ' -e blah' as a very simple workaround. Simply lstrip() the option to put it back to normal, if you like.

Or, if the first "sub-argument" is not also a valid argument to the original function then you shouldn't need to do anything at all. That is, the only reason that python tst.py -e '-s hi -e blah' doesn't work is because -s is a valid option to tst.py.

Also, the optparse module, now deprecated, works without any issue.

Unpeople

I don't think it happens because -s is a valid option to the subparser. I tried it with python Application.py queue -e "-notarealoption" and got the same error. I like using lstrip a little better than replace + with - like SethMMorton suggested though, but it really seems like there should be a way to quote a string such that nothing inside it is replaced/altered/read by argparse. – Shanika

Oh really? I had just based that supposition off my short testing right now. I made a script that took one argument, -a, and simply sent it -a '-b hello' and it worked just fine. But I am using a different version of Python, I guess. – Unpeople

I edited my original question. Apparently this is a known bug in argparse in >2.7 :(. I ended up altering sys.argv before I called parser.parse_args() to add a dummy character to the beginning of the -env option, and stripping it off afterwards. It's hacky and unpythonic as hell but I finally got what I wanted. – Shanika

I had success using a string in quotes that begins with a space. So my parser failed on an argument multi-value input of -10:a 10:b but worked for ' -10:a' 10:b. – Sloatman

I can't believe this package suck at handling a simple dash as argument. Django admin still use it, so I need to use this. If not, I can just use Click – Rickety

@William: -a '-b hello' does work; but try -a '-bhello', it will break regardless if you have -b or -bhello options. It looks like if your quoted value has space in it, argparse can handle it fine even if it starts with dash. If not, not so much. – Changeover

Answer 2

Updated answer:

You can put an equals sign when you call it:

python Application.py -env="-env"

---

Original answer:

I too have had troubles doing what you are trying to do, but there is a workaround build into argparse, which is the parse_known_args method. This will let all arguments that you haven't defined pass through the parser with the assumption that you would use them for a subprocess. The drawbacks are that you won't get error reporting with bad arguments, and you will have to make sure that there is no collision between your options and your subprocess's options.

Another option could be to force the user's to use a plus instead of a minus:

python Application.py -e "+s WHATEVER +e COOL STUFF"

and then you change the '+' to '-' in post processing before passing to your subprocess.

Gloriole

I don't think parse_known_args helps me. I'm not looking to read the arguments in the quotes at all; I'd like the quoted string to be passed as a single object to -env. I've considered going the post processing route, and I probably will if I don't get a better answer from here, but it feels hacky, and it means that + characters in the string are changed to -. I'd really like to be able to pass a string with any characters in it at all. – Shanika

I see what you are asking... If you want to read in multiple strings without the quotes then use nargs='+' which tells -env to read in one or more strings. – Gloriole

But I'd also like some of those strings to have dashes in them, even possibly having the same names as the arguments in my subparser. Something like python Application.py queue -env "-env blah" should work. – Shanika

I'm sorry, I'm out of ideas. I've tried to do the same thing but ultimately opted to simply reimplement the options in my parser to pass the the subprocess because I couldn't get what you are trying to do to work. Good luck! I hope someone comes up with a good suggestions we can try. – Gloriole

@sfendel Try using an equals sign: python Application.py -env="-env" – Gloriole

@Gloriole python Application.py -env="-env" works perfectly for me thanks! You should consider posting it as a new answer so it's not buried down here. – Erdda

Answer 3

This issue is discussed in depth in http://bugs.python.org/issue9334. Most of the activity was in 2011. I added a patch last year, but there's quite a backlog of argparse patches.

At issue is the potential ambiguity in a string like '--env', or "-s WHATEVER -e COOL STUFF" when it follows an option that takes an argument.

optparse does a simple left to right parse. The first --env is an option flag that takes one argument, so it consumes the next, regardless of what it looks like. argparse, on the other hand, loops through the strings twice. First it categorizes them as 'O' or 'A' (option flag or argument). On the second loop it consumes them, using a re like pattern matching to handle variable nargs values. In this case it looks like we have OO, two flags and no arguments.

The solution when using argparse is to make sure an argument string will not be confused for an option flag. Possibilities that have been shown here (and in the bug issue) include:

--env="--env"  # clearly defines the argument.

--env " --env"  # other non - character
--env "--env "  # space after

--env "--env one two"  # but not '--env "-env one two"'

By itself '--env' looks like a flag (even when quoted, see sys.argv), but when followed by other strings it does not. But "-env one two" has problems because it can be parsed as ['-e','nv one two'], a `'-e' flag followed by a string (or even more options).

-- and nargs=argparse.PARSER can also be used to force argparse to view all following strings as arguments. But they only work at the end of argument lists.

There is a proposed patch in issue9334 to add a args_default_to_positional=True mode. In this mode, the parser only classifies strings as option flags if it can clearly match them with defined arguments. Thus '--one' in '--env --one' would be classed as as an argument. But the second '--env' in '--env --env' would still be classed as an option flag.

---

Expanding on the related case in

Using argparse with argument values that begin with a dash ("-")

parser = argparse.ArgumentParser(prog="PROG")
parser.add_argument("-f", "--force", default=False, action="store_true")
parser.add_argument("-e", "--extra")
args = parser.parse_args()
print(args)

produces

1513:~/mypy$ python3 stack16174992.py --extra "--foo one"
Namespace(extra='--foo one', force=False)
1513:~/mypy$ python3 stack16174992.py --extra "-foo one"
usage: PROG [-h] [-f] [-e EXTRA]
PROG: error: argument -e/--extra: expected one argument
1513:~/mypy$ python3 stack16174992.py --extra "-bar one"
Namespace(extra='-bar one', force=False)
1514:~/mypy$ python3 stack16174992.py -fe one
Namespace(extra='one', force=True)

The "-foo one" case fails because the -foo is interpreted as the -f flag plus unspecified extras. This is the same action that allows -fe to be interpreted as ['-f','-e'].

If I change the nargs to REMAINDER (not PARSER), everything after -e is interpreted as arguments for that flag:

parser.add_argument("-e", "--extra", nargs=argparse.REMAINDER)

All cases work. Note the value is a list. And quotes are not needed:

1518:~/mypy$ python3 stack16174992.py --extra "--foo one"
Namespace(extra=['--foo one'], force=False)
1519:~/mypy$ python3 stack16174992.py --extra "-foo one"
Namespace(extra=['-foo one'], force=False)
1519:~/mypy$ python3 stack16174992.py --extra "-bar one"
Namespace(extra=['-bar one'], force=False)
1519:~/mypy$ python3 stack16174992.py -fe one
Namespace(extra=['one'], force=True)
1520:~/mypy$ python3 stack16174992.py --extra --foo one
Namespace(extra=['--foo', 'one'], force=False)
1521:~/mypy$ python3 stack16174992.py --extra -foo one
Namespace(extra=['-foo', 'one'], force=False)

argparse.REMAINDER is like '*', except it takes everything that follows, whether it looks like a flag or not. argparse.PARSER is more like '+', in that it expects a positional like argument first. It's the nargs that subparsers uses.

This uses of REMAINDER is documented, https://docs.python.org/3/library/argparse.html#nargs

Lui

Thank you very much: nargs=argparse.PARSER helped me. – Ptero

I'm not quite sure what's changed, but now python Application.py queue -env "-s WHATEVER -e COOL STUFF" works. queue -env "-foo" still raises the error, because the standalone '-foo' (or '--foo') is still interpreted as a flag. The spaces after the apprent flag do make a difference. – Lui

Answer 4

I have ported a script from optparse to argparse, where certain arguments took values that could start with a negative number. I ran into this problem because the script is used in many places without using the '=' sign to join negative values to the flag. After reading the discussion here and in http://bugs.python.org/issue9334, I know the arguments only take one value and there was no risk in accepting a succeeding argument (ie, a missing value) as the value. FWIW, my solution was to preprocess the arguments and join the problematic ones with '=' before passing to parse_args():

def preprocess_negative_args(argv, flags=None):
    if flags is None:
        flags = ['--time', '--mtime']
    result = []
    i = 0
    while i < len(argv):
        arg = argv[i]
        if arg in flags and i+1 < len(argv) and argv[i+1].startswith('-'):
            arg = arg + "=" + argv[i+1]
            i += 1
        result.append(arg)
        i += 1
    return result

This approach at least does not require any user changes, and it only modifies the arguments which explicitly need to allow negative values.

>>> import argparse
>>> parser = argparse.ArgumentParser("prog")
>>> parser.add_argument("--time")
>>> parser.parse_args(preprocess_negative_args("--time -1d,2".split()))
Namespace(time='-1d,2')

It would be more convenient to tell argparse which arguments should explicitly allow values with a leading dash, but this approach seems like a reasonable compromise.

Celanese

Answer 5

Similar problem. And I solve this by replace space by "\ ". For example:

replace
python Application.py "cmd -option"
by
python Application.py "cmd\ -option".
Not sure for your problem.

Nissy

Answer 6

paser.add_argument("--argument_name", default=None, nargs=argparse.REMAINDER)

python_file.py --argument_name "--abc=10 -a=1 -b=2 cdef"

Note: the argument values have to be passed only within double quotes and this doesn't work with single quotes

Typography

Answer 7

To bypass having to deal with argparse even looking at a '-' for something that isn't a flag you want, you can edit sys.argv before argparse reads it. Just save the argument that you don't want seen, put a filler argument in it's place, and then replace the filler with the original after argparse process sys.argv. I just had to do this for my own code. It's not pretty, but it works and it's easy. You could also use a for loop to iterate through sys.argv if your flags aren't always in the same order.

parser.add_argument('-n', '--input', nargs='*')
spot_saver = ''
if sys.argv[1] == '-n':             #'-n' can be any flag you use
    if sys.argv[2][0] == '-':       #This checks the first character of the element
        spot_saver = sys.argv[2] 
        sys.argv[2] = "fillerText" 
args = parser.parse_args()
if args.input[0] == 'fillerText':
    args.input[0] = spot_saver

Granular

Answer 8

not sure how about newer Python versions but in 3.6.8 I did something like:

parser.add_argument('--eat-args',
                    type=str, 
                    nargs='+',
                    action=EatArgsAction)

where EatArgsAction looks like

class ToDistCpAction(argparse.Action):
  """
  Turns all `"` or `'` into `` (nothing).
  """
  def __call__(self, parser, namespace, values, option_string) -> None:
    setattr(namespace, self.dest, ' '.join(map(lambda arg: re.sub(r'(?<!\w)(\+)', '-', arg), values)))

and this can eat args passed like: ++eat-args +p +I +whatever+anything

you will get --eat-args -p -I -whatever+anything

EDIT: change after @tripleee rebuke

Katiakatie

That's just weird, and not what the question asks about. The shell will remove any syntactic quotes; any remaining quotes will have been put there on purpose by the user. Why would you filter those out? – Infertile

@Infertile you're right, I didn't check it. Just noticed that it is applicable in case when you want to pass these args in for example .properties file for Oozie job. – Katiakatie