Calling a Python function with *args,**kwargs and optional / default arguments

Question

In Python I can define a function as follows:

def func(kw1=None,kw2=None,**kwargs):
   ...

In this case, I can call func as:

func(kw1=3,kw2=4,who_knows_if_this_will_be_used=7,more_kwargs=Ellipsis)

I can also define a function as:

def func(arg1,arg2,*args):
    ...

which can be called as

func(3,4,additional,arguments,go,here,Ellipsis)

Finally, I can combine the two forms

def func(arg1,arg2,*args,**kwargs):
    ...

But, what does not work is calling:

func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs):  #SYNTAX ERROR (in Python 2 only, apparently this works in Python 3)
    ...

My original thought was that this was probably because a function

def func(arg1,arg2,*args,kw1=None):
    ...

can be called as

func(1,2,3)  #kw1 will be assigned 3

So this would introduce some ambiguity as to whether 3 should be packed into args or kwargs. However, with Python 3, there is the ability to specify keyword only arguments:

def func(a,b,*,kw=None):  # can be called as func(1,2), func(1,2,kw=3), but NOT func(1,2,3)
    ...

With this, it seems that there is no syntactic ambiguity with:

def func(a,b,*args,*,kw1=None,**kwargs):
    ...

However, this still brings up a syntax error (tested with Python3.2). Is there a reason for this that I am missing? And, is there a way to get the behavior I described above (Having *args with default arguments) -- I know I can simulate that behavior by manipulating the kwargs dictionary inside the function.

Serif

no idea why you got syntax error for "def func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs):" – Payer

@Payer Because I didn't test that version on python 3, only python 2. I just assumed the final version would work, and when It didn't, I assumed the previous versions wouldn't work either. Thanks!. – Serif

How should a bare 3 ever get into kwargs? What keyword would it use? I cannot see any ambiguity. Note that the bare * in the argument list is only useful if there's no *args. It's a placeholder you use instead of *args. – Millepore

Good question but why spreading the world-encompassing confusion of calling parameters (those guys in function definitions) arguments (those fellows in function calls)? – Savagery

Answer 1

You can do that in Python 3.

def func(a,b,*args,kw1=None,**kwargs):

The bare * is only used when you want to specify keyword only arguments without accepting a variable number of positional arguments with *args. You don't use two *s.

To quote from the grammar, in Python 2, you have

parameter_list ::=  (defparameter ",")*
                    (  "*" identifier [, "**" identifier]
                    | "**" identifier
                    | defparameter [","] )

while in Python 3, you have

parameter_list ::=  (defparameter ",")*
                    (  "*" [parameter] ("," defparameter)*
                    [, "**" parameter]
                    | "**" parameter
                    | defparameter [","] )

which includes a provision for additional parameters after the * parameter.

UPDATE:

Latest Python 3 documentation here.

Pryor

*args should be after keyword arguments should it not? e.g. def func(a,b, kw1=None, *args, **kwargs): – Samul

@Nimi I'm specifically showing a Python 3 only feature that allows you to put named arguments after *args, so they can only be used by name, not position. What you're showing allows kw1 to be filled in by position or name -- for your version func(1, 2, 3) will fill in a, b, and kw1, where in my version func(1, 2, 3) will fill in a, b, and args, and require you to do func(1, 2, kw1=3) if you want to fill in kw1. – Pryor

Answer 2

If you want to do a mixture of both remember that *args and **kwargs must be the last parameters specified.

def func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs): #Invalid
def func(arg1,arg2,kw1=None,kw2=None,*args,**kwargs): #Valid

The comments seem to be based on mixing up how a function definition is constructed compared to how the arguments provided are assigned back to the parameters specified in the definition.

This is the definition of this function which has 6 parameters. It is called by passing named and unnamed arguments to it in a function call.

For this example... When an argument is named when calling the function it can be provided out of order. arg1 and arg2 are mandatory parameters and if not passed to the function as named arguments, then they must be assigned in order from the provided unnamed arguments. kw1 and kw2 have default values provided in the function definition so they are not mandatory, but if not provided for as named arguments they will take any available values from the remaining provided unnamed arguments. Any unnamed arguments left over are provided to the function in an array called args Any named arguments that do not have a corresponding parameter name in the function definition are provided to the function call in a dictionary called kwargs.

Esthete

This is syntactically valid, but doesn't work as one might expect because there's ambiguity as to whether extra required arguments are mapped to the named keyword arguments or *args. – Cristincristina

I would go a step further and say that this does not work for making kw1 and kw2 'keyword' args with Python 2.7, which is why I'm here! The form would be better listed as func(arg1,arg2,arg3=None,arg4=None,*args,**kwargs): #Valid with defaults on positional args, but this is really just four positional args, two of which are optional. To pass kwargs, you will need to fill in all four args, including arg3 and arg4. – Shopkeeper

I suggest https://mcmap.net/q/169123/-default-arguments-with-args-and-kwargs for Python 2.7 and notice the 'def_val'. – Shopkeeper

As @Shopkeeper mentioned, this means you NEED to fill in kw1 and kw2 if you want to define args* – Joachim

This is only true for Python 2, not Python 3, where arguments specified after the *args var-positional parameter (or a lone *) are called keyword-only parameters. These do not even need to have default values. – Help

Answer 3

Clear and concise:

In Python 3.5 or greater:

def foo(a, b=3, *args, **kwargs):
  defaultKwargs = { 'c': 10, 'd': 12 }
  kwargs = { **defaultKwargs, **kwargs }
  
  print(a, b, args, kwargs)
  
  # Do something else

foo(1) # 1 3 () {'c': 10, 'd': 12}
foo(1, d=5) # 1 3 () {'c': 10, 'd': 5}
foo(1, 2, 4, d=5) # 1 2 (4,) {'c': 10, 'd': 5}

Note: you can use In Python 2

kwargs = merge_two_dicts(defaultKwargs, kwargs)

In Python 3.5

kwargs = { **defaultKwargs, **kwargs }

In Python 3.9

kwargs = defaultKwargs | kwargs  # NOTE: 3.9+ ONLY

Spode

Answer 4

If you are looking to do that in Python 2, I have found a workaround explained in this post, using a decorator.

This decorator assigns default kwarg values if they are not strictly defined.

from functools import wraps

def force_kwargs(**defaultKwargs):
    def decorator(f):
        @wraps(f)
        def g(*args, **kwargs):
            new_args = {}
            new_kwargs = defaultKwargs
            varnames = f.__code__.co_varnames
            new_kwargs.update(kwargs)
            for k, v in defaultKwargs.items():
                if k in varnames:
                    i = varnames.index(k)
                    new_args[(i, k)] = new_kwargs.pop(k)
            # Insert new_args into the correct position of the args.
            full_args = list(args)
            for i, k in sorted(new_args.keys()):
                if i <= len(full_args):
                    full_args.insert(i, new_args.pop((i, k)))
                else:
                    break
            # re-insert the value as a key-value pair
            for (i, k), val in new_args.items():
                new_kwargs[k] = val
            return f(*tuple(full_args), **new_kwargs)
        return g
    return decorator

Result

@force_kwargs(c=7, z=10)
def f(a, b='B', c='C', d='D', *args, **kw):
    return a, b, c, d, args, kw
#                                    a    b  c    d  args      kwargs
f('r')                           # 'r', 'B', 7, 'D',    (),       {'z': 10}
f(1, 2, 3, 4, 5)                 #   1,   2, 7,   3, (4,5),       {'z': 10}
f(1, 2, 3, b=0, c=9, f='F', z=5) #   1,   0, 9,   2,  (3,), {'f': 'F', 'z': 5}

Variant

If you want to use the default values as written in the function definition, you could access the argument default values using f.func_defaults, which lists the default values. You would have to zip them with the end of the f.__code__.varnames to match these default values with the variable names.

Gazebo