Create variable key/value pairs with argparse (python)

Question

I'm using argparse module to set my command line options. I'm also using a dict as a config in my application. Simple key/value store.

What I'm looking for is a possibility to override JSON options using command line arguments, without defining all possible arguments in advance. Something like --conf-key-1 value1 --conf-key-2 value2, which would create a dict {'key_1': 'value1','key_2': 'value2'} ('-' in the argument is replaced by '_' in the dict). Then I can combine this dict with my JSON config (dict).

So basically I would like to define --conf-* as an argument, where * can be any key and what comes after is the value.

I did find configargparse module, but as far as I can see I start with a dict I already use.

Any ideas how I could approach this?

Eulogistic

related: #45025914 – Eurypterid

https://mcmap.net/q/86530/-what-is-the-right-way-to-treat-python-argparse-namespace-as-a-dictionary – Inerasable

Answer 1

The first thing I'd try is use parse_known_args to handle other arguments, and handle the list of extras with my on routine. Adding the '--conf-' handling to argparse would be more work.

argv = '--conf-key-1 value1 --conf-key-2 value2'.split()
p = argparse.ArgumentParser()
args, extras = p.parse_known_args(argv)

def foo(astr):
    if astr.startswith('--conf-'):
        astr = astr[7:]
    astr = astr.replace('-','_')
    return astr

d = {foo(k):v for k,v in zip(extras[::2],extras[1::2])}
# {'key_1': 'value1', 'key_2': 'value2'}

The extras parsing could be more robust - making sure that there are proper pairs, rejecting badly formed keys, handling =.

Another approach would be to scan sys.argv for --conf- strings, and use those to construct add_argument statements.

keys = [k for k in argv if k.startswith('--conf-')]
p = argparse.ArgumentParser()
for k in keys:
    p.add_argument(k, dest=foo(k))
print vars(p.parse_args(argv))

---

If you would accept '--conf key1 value1 --conf key2 value2 ...' as the input, you could define

parser.add_argument('--conf', nargs=2, action='append')

which would produce:

namespace('conf': [['key1','value1'],['key2','value2']])

which could easily be turned into a dictionary. Or a custom Action could use setattr(namespace, values[0], values[1]) to enter the key/value pairs directly into the namespace.

I believe there have been SO question(s) about accepting '"key1:value" "key2:value2"' inputs.

Hostler

Sound like a way :)If there is no other way, I will try this. I still hope there is a 'clean' solution within argparse. – Eulogistic

There's no way in argparse of defining an options pattern. It can look for abbreviations and aliases, but not patterns. optparse has a custom action mechanism that might be able to handle this. – Hostler

Answer 2

I had a similar issue and found a very workable pattern that works well with argparse (here three key-pairs: foo, bar and baz:

mycommand par1 --set foo=hello bar="hello world" baz=5

1. Defining the optional, multivalued argument

The set argument must be defined so:

import argparse
parser = argparse.ArgumentParser(description="...")
...
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+',
                        help="Set a number of key-value pairs "
                             "(do not put spaces before or after the = sign). "
                             "If a value contains spaces, you should define "
                             "it with double quotes: "
                             'foo="this is a sentence". Note that '
                             "values are always treated as strings.")
args = parser.parse_args()

The argument is optional and multivalued, with a minimum of one occurrence (nargs='+').

The result is a list of strings e.g. ["foo=hello", "bar=hello world", "baz=5"] in args.set, which we now need to parse (note how the shell has processed and removed the quotes!).

2. Parsing the result

For this we need 2 helper functions:

def parse_var(s):
    """
    Parse a key, value pair, separated by '='
    That's the reverse of ShellArgs.

    On the command line (argparse) a declaration will typically look like:
        foo=hello
    or
        foo="hello world"
    """
    items = s.split('=')
    key = items[0].strip() # we remove blanks around keys, as is logical
    if len(items) > 1:
        # rejoin the rest:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    """
    Parse a series of key-value pairs and return a dictionary
    """
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

At this point it is very simple:

# parse the key-value pairs
values = parse_vars(args.set)

You now have a dictionary:

values = {'foo':'hello', 'bar':'hello world', 'baz':'5'}

Note how the values are always returned as strings.

This method is also documented as a git gist.

Handtohand

thanks for this! I just want to add that there's a simpler way IMO to parse the pairs dict(map(lambda s: s.split('='), args.set)) – Grantley

Very elegant solution, indeed! In your solution, I might keep the parse_var function instead of s.split('=') to make sure it takes case of all border cases, e.g. one of the keys starts with a whitespace, etc. In practice, I also made it raise an error if no = was found. – Handtohand

I think that the first function has a bug: If len(items) <= 1, value will be undefined – Bluff

Answer 3

This can all be done much more simply using str.split(delim, limit):

class kvdictAppendAction(argparse.Action):
    """
    argparse action to split an argument into KEY=VALUE form
    on the first = and append to a dictionary.
    """
    def __call__(self, parser, args, values, option_string=None):
        assert(len(values) == 1)
        try:
            (k, v) = values[0].split("=", 2)
        except ValueError as ex:
            raise argparse.ArgumentError(self, f"could not parse argument \"{values[0]}\" as k=v format")
        d = getattr(args, self.dest) or {}
        d[k] = v
        setattr(args, self.dest, d)

...


myparser.add_argument("--keyvalue",
                      nargs=1,
                      action=kvdictAppendAction,
                      metavar="KEY=VALUE",
                      help="Add key/value params. May appear multiple times.")

Sulcate

Answer 4

here and there partial answers. Let's sum up to the standard argparse way.

import argparse
class kwargs_append_action(argparse.Action):
"""
argparse action to split an argument into KEY=VALUE form
on append to a dictionary.
"""

def __call__(self, parser, args, values, option_string=None):
    try:
        d = dict(map(lambda x: x.split('='),values))
    except ValueError as ex:
        raise argparse.ArgumentError(self, f"Could not parse argument \"{values}\" as k1=v1 k2=v2 ... format")
    setattr(args, self.dest, d)

parser = argparse.ArgumentParser(description="...")
parser.add_argument("-f", "--filters", dest="filters",
                        nargs='*',
                        default={'k1':1, 'k2': "P"},
                        required=False,
                        action=kwargs_append_action,
                        metavar="KEY=VALUE",
                        help="Add key/value params. May appear multiple times. Aggregate in dict")
args = parser.parse_args()

and the usage:

python main.py --filters foo=1 bar="coucou"

Theft

Answer 5

Current code for the top answer has a bug where if the "items" variable in the "parse_var" function doesn't have an "=" it returns the variable "value" before assignment. Here is a modified / condensed function that provides some error handling for whether an "=" was provided in each argument:

def parse_vars(items):
    """
        Parse a series of key-value pairs and return a dictionary and 
        a success boolean for whether each item was successfully parsed.
    """
    count = 0
    d = {}
    for item in items:
        if "=" in item:
            split_string = item.split("=")
            d[split_string[0].strip()] = split_string[1].strip()
            count += 1
        else:
            print(f"Error: Invalid argument provided - {item}")
        
    return d, count == len(items)

Kennie

Answer 6

To simplify slightly on fralaus answer, the 2 methods can be combined into one easily.

Note: My docstring etc differ as I was using it for ansible extra_vars, but the core logic for string splitting came from fralaus' answer.

 def parse_vars(extra_vars):
     """
     Take a list of comma seperated key value pair strings, seperated
     by comma strings like 'foo=bar' and return as dict.
     :param extra_vars: list[str] ['foo=bar, 'key2=value2']

     :return: dict[str, str] {'foo': 'bar', 'key2': 'value2'}
     """
     vars_list = []
     if extra_vars:
         for i in extra_vars:
            items = i.split('=')
            key = items[0].strip()
            if len(items) > 1:
                value = '='.join(items[1:])
                vars_list.append((key, value))
     return dict(vars_list)

print parse_vars(args.set)

 $ test.py --set blah=gar one=too
>> {"blah": "gar", "one": "too"}

Scarabaeus

Answer 7

This answer is based on other modifications of fralau's answer with Action implementation, but given --set A=a B=b C=c1 --set D=d C=c2 will merge the pairs and return {'A':'a', 'B':'b', 'C':'c2', 'D':'d'}

class MergeKeyValuePairs(argparse.Action):
   """
   argparse action to split a KEY=VALUE argument and append the pairs to a dictionary.
   """
   def __call__(self, parser, args, values, option_string=None):
      previous = getattr(args, self.dest, None) or dict()
      try:
         added = dict(map(lambda x: x.split('='), values))
      except ValueError as ex:
         raise argparse.ArgumentError(self, f"Could not parse argument \"{values}\" as k1=v1 k2=v2 ... format")
      merged = {**previous, **added}
      setattr(args, self.dest, merged)

And then as usual:

myparser.add_argument("--set",
                      nargs=*,
                      action=MergeKeyValuePairs,
                      metavar="KEY=VALUE",
                      help='Add key-value mappings. Do not place spaces before or after the = sign.'
                           ' If a value contains spaces, please wrap it in double quotes: key="value with spaces".')

Stacistacia

Answer 8

found a very simple solution : vars(args) will transform it into a dict object.

Inerasable