type=dict in argparse.add_argument()

Question

I want to use the standard library argparse module to parse command line arguments to my program, and have the program accept an optional argument -i (or --image) which is a dictionary.

I tried configuring the parser like this:

parser.add_argument('-i','--image', type=dict, help='Generate an image map from the input file (syntax: {\'name\': <name>, \'voids\': \'#08080808\', \'0\': \'#00ff00ff\', \'100%%\': \'#ff00ff00\'}).')

But when I try to run the script, I get an error:

$ ./script.py -i {'name': 'img.png','voids': '#00ff00ff','0': '#ff00ff00','100%': '#f80654ff'}
    
script.py: error: argument -i/--image: invalid dict value: '{name:'

Even though similar syntax would work fine inside the interpreter:

>>> a={'name': 'img.png','voids': '#00ff00ff','0': '#ff00ff00','100%': '#f80654ff'}

How should I a) write the command line and b) set up the argparse logic?

Lambeth

You can read formats like JSON from external file or stdin, and then parse it. So you argparse type will be actually a file. – Jolo

as @wim said in his answer the shell is processing the arguments before passing them down to python. If you prepend your command with 'echo' (echo ./script.py -i {'name': ...) you'll see what python is seeing (mainly it is not receiving any quotes). In your case that there is no $ in your param (that could be interpreted by the shell as an enviroment variable) you can surround your dict with double quotes: ./script.py -i "{'name': 'img.png', ....}" – Seattle

Trying to pass a dict on the command line - using Python-like syntax - is really not how the command line is intended to work. It will be difficult for users to get the syntax right, because of how the shell tokenizes the command line to figure out separate arguments. It doesn't know anything about balancing brackets, only quoting and escaping - and different shells/terminals/operating systems have different rules for how that works. – Verdha

Answer 1

Necroing this: json.loads works here, too. It doesn't seem too dirty.

import json
import argparse

test = '{"name": "img.png","voids": "#00ff00ff","0": "#ff00ff00","100%": "#f80654ff"}'

parser = argparse.ArgumentParser()
parser.add_argument('-i', '--input', type=json.loads)

args = parser.parse_args(['-i', test])

print(args.input)

Returns:

{u'0': u'#ff00ff00', u'100%': u'#f80654ff', u'voids': u'#00ff00ff', u'name': u'img.png'}

Hardecanute

json.loads is a nice choice for type. Like int and float it takes a string, and returns a ValueError if it can't handle it. It is also safer than eval. For this purpose it may be a little too general (i.e. it can handle a list '[1, 2]'), but the user can deal with that after the parse_args(). – Ecchymosis

When the values are strings, int, float, it works fine. For other types of values, such as bool, it does not (but passing 1 for True should work but all well-written code). – Context

can you add an example of the cli input? almost everything I try results in invalid loads value. e.g this works,. --input="{}" this fails --input="{'foo': 'bar'}" – Gary

@J'e --input='{"foo" : "bar" }' will work, because JSON Syntax mandates double quotes for string. – Antecedency

It's failed when I use python3 main.py --input='{}' instead of args = parser.parse_args(['-i', test]) – Cavit

Answer 2

For completeness, and similarly to json.loads, you could use yaml.load (available from PyYAML in PyPI). This has the advantage over json in that there is no need to quote individual keys and values on the command line unless you are trying to, say, force integers into strings or otherwise overcome yaml conversion semantics. But obviously the whole string will need quoting as it contains spaces!

>>> import argparse
>>> import yaml
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('-fna', '--filename-arguments', type=yaml.load)
>>> data = "{location: warehouse A, site: Gloucester Business Village}"
>>> ans = parser.parse_args(['-fna', data])
>>> print ans.filename_arguments['site']
Gloucester Business Village

Although admittedly in the question given, many of the keys and values would have to be quoted or rephrased to prevent yaml from barfing. Using the following data seems to work quite nicely, if you need numeric rather than string values:

>>> parser.add_argument('-i', '--image', type=yaml.load)
>>> data = "{name: img.png, voids: 0x00ff00ff, '0%': 0xff00ff00, '100%': 0xf80654ff}"
>>> ans = parser.parse_args(['-i', data])
>>> print ans.image
{'100%': 4161164543L, 'voids': 16711935, 'name': 'img.png', '0%': 4278255360L}

Theretofore

Corrected (lack of) parser.parse_args calls. Thanks for pointing this out, Hotschke – Theretofore

Got to use yaml.safe_load now, or otherwise pass a Loader argument using functools.partial – Auvil

Answer 3

Using simple lambda parsing is quite flexible:

parser.add_argument(
    '--fieldMap',
    type=lambda x: {k:int(v) for k,v in (i.split(':') for i in x.split(','))},
    help='comma-separated field:position pairs, e.g. Date:0,Amount:2,Payee:5,Memo:9'
)

Haplosis

That's the cleanest solution so far! Thanks, helped a lot – Virilism

Very clean solution, Thanks. As v is used in multiple context, its creating confusion for me, below is working as well type=lambda e: {k:int(v) for k,v in (x.split(':') for x in e.split(','))}, – Bowens

Answer 4

I’ll bet your shell is messing with the braces, since curly braces are the syntax used for brace expansion features in many shells (see here).

Passing in a complex container such as a dictionary, requiring the user to know Python syntax, seems a bad design choice in a command line interface. Instead, I’d recommend just passing options in one-by-one in the CLI within an argument group, and then build the dict programmatically from the parsed group.

Unmoving

Changed in version 3.11: Calling add_argument_group() on an argument group is deprecated. This feature was never supported and does not always work correctly. The function exists on the API by accident through inheritance and will be removed in the future. As per the API docs, can be referred below in more details docs.python.org/dev/library/… – Corporal

@AnuragUpadhyaya So what? This answer suggests to add an argument group on the parser, not on a returned group. The feature itself isn't being removed, only an unrelated misuse of the feature.. – Unmoving

This seems to be the only answer that actually addresses the reported error (which is caused by the command line rather than by argparse - although the argparse usage is also wrong). – Verdha

Answer 5

Combining the type= piece from @Edd and the ast.literal_eval piece from @Bradley yields the most direct solution, IMO. It allows direct retrieval of the argval and even takes a (quoted) default value for the dict:

Code snippet

parser.add_argument('--params', '--p', help='dict of params ', type=ast.literal_eval, default="{'name': 'adam'}")
args = parser.parse_args()

Running the Code

python test.py --p "{'town': 'union'}"

note the quotes on the dict value. This quoting works on Windows and Linux (tested with [t]csh).

Retrieving the Argval

dict=args.params

Exegetic

Not sure if this is something that has changed, but i'm using python3.9.12 and I had to use literal_eval not literal.eval – Groundmass

You are correct - I mistyped original! I'll edit to correct. – Exegetic

Answer 6

You can definitely get in something that looks like a dictionary literal into the argument parser, but you've got to quote it so when the shell parses your command line, it comes in as

• a single argument instead of many (the space character is the normal argument delimiter)
• properly quoted (the shell removes quotes during parsing, because it's using them for grouping)

So something like this can get the text you wanted into your program:

python MYSCRIPT.py -i "{\"name\": \"img.png\", \"voids\": \"#00ff00ff\",\"0\": \"#ff00ff00\",\"100%\": \"#f80654ff\"}"

However, this string is not a valid argument to the dict constructor; instead, it's a valid python code snippet. You could tell your argument parser that the "type" of this argument is eval, and that will work:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('-i','--image', type=eval, help='Generate an image map...')
args = parser.parse_args()
print args

and calling it:

% python MYSCRIPT.py -i "{\"name\": \"img.png\", \"voids\": \"#00ff00ff\",\"0\": \"#ff00ff00\",\"100%\": \"#f80654ff\"}"
Namespace(image={'0': '#ff00ff00', '100%': '#f80654ff', 'voids': '#00ff00ff', 'name': 'img.png'})

But this is not safe; the input could be anything, and you're evaluating arbitrary code. It would be equally unwieldy, but the following would be much safer:

import argparse
import ast

parser = argparse.ArgumentParser()
parser.add_argument('-i','--image', type=ast.literal_eval, help='Generate an image map...')
args = parser.parse_args()
print args

This also works, but is MUCH more restrictive on what it will allow to be eval'd.

Still, it's very unwieldy to have the user type out something, properly quoted, that looks like a python dictionary on the command line. And, you'd have to do some checking after the fact to make sure they passed in a dictionary instead of something else eval-able, and had the right keys in it. Much easier to use if:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--image-name", required=True)
parser.add_argument("--void-color", required=True)
parser.add_argument("--zero-color", required=True)
parser.add_argument("--full-color", required=True)

args = parser.parse_args()

image = {
    "name": args.image_name,
    "voids": args.void_color,
    "0%": args.zero_color,
    "100%": args.full_color
    }
print image

For:

% python MYSCRIPT.py --image-name img.png --void-color \#00ff00ff --zero-color \#ff00ff00 --full-color \#f80654ff
{'100%': '#f80654ff', 'voids': '#00ff00ff', 'name': 'img.png', '0%': '#ff00ff00'}

Knotty

Wow, thanks for the possibilities overview; however, despite in the example I've put only 0 and 100%, these could actually be any value (e.g. {'46%':'#0f0e0d0c','3629','#f0e0d0c0'}), which is not contemplated in your last piece of code... – Lambeth

Answer 7

One of the simplest ways I've found is to parse the dictionary as a list, and then convert that to a dictionary. For example using Python3:

#!/usr/bin/env python3
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('-i', '--image', type=str, nargs='+')
args = parser.parse_args()
if args.image is not None:
    i = iter(args.image)
    args.image = dict(zip(i, i))
print(args)

then you can type on the command line something like:

./script.py -i name img.png voids '#00ff00ff' 0 '#ff00ff00' '100%' '#f80654ff'

to get the desired result:

Namespace(image={'name': 'img.png', '0': '#ff00ff00', 'voids': '#00ff00ff', '100%': '#f80654ff'})

Chancellery

Answer 8

A minimal example to pass arguments as a dictionary from the command line:

# file.py
import argparse
import json
parser = argparse.ArgumentParser()
parser.add_argument("-par", "--parameters",
                    required=False,
                    default=None,
                    type=json.loads
                )
args = parser.parse_args()
print(args.parameters)

and in the terminal you can pass your arguments as a dictionary using a string format:

python file.py --parameters '{"a":1}'

Brighten

Answer 9

General Advice: DO NOT USE eval.

If you really have to ... "eval" is dangerous. Use it if you are sure no one will knowingly input malicious input. Even then there can be disadvantages. I have covered one bad example.

Using eval instead of json.loads has some advantages as well though. A dict doesn't really need to be a valid json. Hence, eval can be pretty lenient in accepting "dictionaries". We can take care of the "danger" part by making sure that final result is indeed a python dictionary.

import json
import argparse

tests = [
  '{"name": "img.png","voids": "#00ff00ff","0": "#ff00ff00","100%": "#f80654ff"}',
  '{"a": 1}',
  "{'b':1}",
  "{'$abc': '$123'}",
  '{"a": "a" "b"}' # Bad dictionary but still accepted by eval
]
def eval_json(x):
  dicti = eval(x)
  assert isinstance(dicti, dict)
  return dicti

parser = argparse.ArgumentParser()
parser.add_argument('-i', '--input', type=eval_json)
for test in tests:
  args = parser.parse_args(['-i', test])
  print(args)

Output:

Namespace(input={'name': 'img.png', '0': '#ff00ff00', '100%': '#f80654ff', 'voids': '#00ff00ff'})
Namespace(input={'a': 1})
Namespace(input={'b': 1})
Namespace(input={'$abc': '$123'})
Namespace(input={'a': 'ab'})

Polypeptide

This is extremely dangerous advice. Using eval will (obviously) cause the input from the cmdline to be evaluated as python. This happens before it returns a value so your type checking is too little too late. Also, there is plenty of valid dangerous python which will still return a dict.... The "we can take care of the danger" statement is both inaccurate and dangerous to be spreading as advice. – Ruddie

Okay. Agree. I am changing the language of the answer. – Polypeptide

Answer 10

 You could try:

$ ./script.py -i "{'name': 'img.png','voids': '#00ff00ff','0': '#ff00ff00','100%': '#f80654ff'}"

I haven't tested this, on my phone right now.

Edit: BTW I agree with @wim, I think having each kv of the dict as an argument would be nicer for the user.

Spermine

Answer 11

Here is a another solution since I had to do something similar myself. I use the ast module to convert the dictionary, which is input to the terminal as a string, to a dict. It is very simple.

Code snippet

Say the following is called test.py:

import argparse
import ast

parser = argparse.ArgumentParser()
parser.add_argument('--params', '--p', help='dict of params ',type=str)

options = parser.parse_args()

my_dict = options.params
my_dict = ast.literal_eval(my_dict)
print(my_dict)
for k in my_dict:
  print(type(my_dict[k]))
  print(k,my_dict[k])

Then in the terminal/cmd line, you would write:

Running the code

python test.py --p '{"name": "Adam", "lr": 0.001, "betas": (0.9, 0.999)}'

Output

{'name': 'Adam', 'lr': 0.001, 'betas': (0.9, 0.999)}
<class 'str'>
name Adam
<class 'float'>
lr 0.001
<class 'tuple'>
betas (0.9, 0.999)

Nygaard

Answer 12

TLDR Solution: The simplest and quickest solution is as below:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-par", "--parameters",
                    default={},
                    type=str)
args = parser.parse_args()

In the parser.add_argument function:

1. Use a dictionary object for default object
2. str as the type

Then args.parameters will automatically be converted to a dictionary without any need for ast.literal.eval or json.loads.

Motivation: The methods posted by @Galuoises and @frankeye, appear to not work when the default is set as a json encoded dictionary such as below.

parser.add_argument("-par", "--parameters",
                required=False,  default="{\"k1\":v1, \"k2\":v2}",
                type=json.loads)

This is because

Groom

Answer 13

The following works just fine:

parser = argparse.ArgumentParser()
parser.add_argument("-par", "--parameters",
                required=False,  default={"k1a":"v1a","k2a":"v2a"},
                type=json.loads)
args = parser.parse_args()
print(str(parameters))

result:
{'k1a': 'v1a', 'k2a': 'v2a'}

For default value, the type should be dict since json.loads returns a dictionary, not a string, the default object should be given as a dictionary.

import argparse,json,sys
sys.argv.extend(['-par','{"k1b":"v1b","k2b":"v2b"}'])
parser = argparse.ArgumentParser()
parser.add_argument("-par", "--parameters",
                required=False,  default={"k1":"v1","k2":"v2"},
                type=json.loads)
args = parser.parse_args()
print(str(args.parameters))

result: 
{'k1b': 'v1b', 'k2b': 'v2b'}

Peridotite

Answer 14

In my case I need to pass list of labels with multiple values and I wanted to get them as dictionary. So I did it with combination of nargs and Action:

class LabelAction(argparse.Action):
    def __call__(self, parser: argparse.ArgumentParser, namespace: argparse.Namespace,
                 values: Union[str, Sequence[Any], None],
                 option_string: Optional[str] = ...) -> None:
        labels = getattr(namespace, self.dest, None) or {}
        labels[values[0]] = values[1:]
        setattr(namespace, self.dest, labels)

parser = argparse.ArgumentParser(description='Labels dict')
parser.add_argument('-l', nargs='*', action=LabelAction)

And the line of arguments:

-l label1 v1 v2 v3 -l label2 x1 x10

brings the dictionary:

{ 
  'label1': ['v1', 'v2', 'v3'],
  'label2': ['x1', 'x10'],
}

Ellon