python pass different **kwargs to multiple functions

Question

From python doc and stackoverflow, I understand how to use the **kwargs in my def function. However, I have a case need two sets of **kwargs for two sub functions. Can someone show me how to separate the **kwargs properly?

Here is my goal: to plot set of points and interpolated smooth curve,
and my naive sample code:

def smoothy(x,y, kind='cubic', order = 3, **kwargs_for_scatter, **kwargs_for_plot):
    yn_cor = interp1d(x, y, kind=kind, assume_sorted = False)
    xn = np.linspace(np.min(x), np.max(x), len(x) * order)
    plt.scatter(x,y, **kwargs_for_scatter)
    plt.plot(xn, yn_cor(xn), **kwargs_for_plot);
    return

Thanks for help.

Gerundive

Just pass scatter and plot as normal dicts? It appears they're required arguments... then use **scatter or **plot in the functions... you're not doing anything else with them – Mutualize

Answer 1

There is no such mechanism. There is a proposal, PEP-448, whereby Python 3.5 and following generalize argument unpacking. Python 3.4 and previous don't support it. Best you can do in general:

def smoothy(x,y, kind='cubic', order = 3, kwargs_for_scatter={}, kwargs_for_plot={}):
    yn_cor = interp1d(x, y, kind=kind, assume_sorted = False)
    xn = np.linspace(np.min(x), np.max(x), len(x) * order)
    plt.scatter(x,y, **kwargs_for_scatter)
    plt.plot(xn, yn_cor(xn), **kwargs_for_plot);
    return

Then pass in those options as dictionaries, not kwargs, to smoothy.

smoothy(x, y, 'cubic', 3, {...}, {...})

Because the variable names would then be possibly exposed to callers, you may want to rename them something shorter (perhaps scatter_options and plot_options).

Update: Python 3.5 and 3.6 are now mainstream, and they indeed support an extended unpacking syntax based on PEP-448.

>>> d = {'name': 'joe'}
>>> e = {'age': 20}
>>> { **d, **e }
{'name': 'joe', 'age': 20}

This does not, however, help much in this kwargs-intended-for-multiple-destinations scenario. Even if the smoothy() function took a unified grab-bag of kwargs, you'd need to determine which of them were intended for which subfunctions. Messy at the very best. Multiple dict parameters, one intended to be passed to each kwarg-taking subfunction, still the best approach.

Noun

As I suggested here no point really keeping the kwargs_for_ in the arguments any more... (imho anyway) – Mutualize

@JonClements Looks like we had a race condition / concurrent editing, but came to the same conclusion! – Noun

No worries... perfectly good answer (although I'm biased cough)... I just wasn't sure if it was that simple or if something more convoluted was required. +1 from me for spelling it out and options :) – Mutualize

Answer 2

Another, more different approach

I realize I am a bit late to the party. However, I stumbled across a similar issue when dealing with a class composed of several other classes. I wanted to avoid passing dictionaries for each sub-class (or -function) and it would be very anti-dry to copy all the arguments of the component classes and additionally run the risk of having to update all of them at a later stage.

My solution is certainly not the shortest nor is it very nice, but I think it has a certain elegance to it. I modified the function smoothy below:

import inspect

def smoothy(x,y, kind='cubic', order = 3, **kwargs):
    yn_cor = interp1d(x, y, kind=kind, assume_sorted = False)
    xn = np.linspace(np.min(x), np.max(x), len(x) * order)
    
    scatter_args = list(inspect.signature(plt.scatter).parameters)
    scatter_dict = {k: kwargs.pop(k) for k in dict(kwargs) if k in scatter_args}
    plt.scatter(x,y, **scatter_dict)
    
    plot_args = list(inspect.signature(plt.plot).parameters)
    plot_dict = {k: kwargs.pop(k) for k in dict(kwargs) if k in plot_args}
    plt.plot(xn, yn_cor(xn), **plot_dict);
    return

Explantion

To start off, make a list (scatter_args) of the arguments that the first function (scatter) accepts, using inspect.signature(). Then construct a new dictionary (scatter_dict) from kwargs, only extracting items that are also in our list of arguments. Using dict(kwargs) here ensures that we loop over a copy of kwargs, so that we can alter the original without running into errors. This new dictionary can then be passed to the function (scatter) and the steps are repeated for the next function.

A pitfall is that argument names in kwargs may not be repeated, since it is now a single dict. So for pre-built functions where you do not control argument names you might run into problems with this method.

This does allow me to then use said composed class as a parent (or sub) class (passing on the remainder of kwargs).

Winded

I like your answer, but instead of looping through the inspect.signature I would do the following: list(inspect.signature(plt.scatter)) this will directly convert it into list – Thorne

@Thorne Thanks for the suggestion, that makes it considerably shorter and more readable. – Winded

Answer 3

Use a Class for Help

I came to this question, because I needed to do something similar. After some thinking, it seemed that a class approach would help me. I hope this can extend to some others too.

import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import interp1d

class KWAs:
    def __init__(self, algo):
        self.algo = algo
        self.kwargs_dict = {
            'scatter_params':{},
            'plot_params':{}
        } # preloading group keys allows plotting when a kwarg group is absent.

    def add_kwargs_to_dict(self, group_name, **kwargs):
        self.kwargs_dict[group_name] = kwargs

    def list_kwargs(self):
        print('Listing all kwarg groups:')
        for kwargs in self.kwargs_dict:
            print('\tkwarg group {}: {}'.format(kwargs, self.kwargs_dict[kwargs]))
        print()

    def get_kwarg_group(self,group):
        print('kwarg group {}: {}'.format(group, self.kwargs_dict[group]))
        print()

    def smoothy(self, x,y, kind='cubic', order = 3):
        yn_cor = interp1d(x, y, kind=kind, assume_sorted = False)
        xn = np.linspace(np.min(x), np.max(x), len(x) * order)
        plt.scatter(x,y, **self.kwargs_dict['scatter_params'])
        plt.plot(xn, yn_cor(xn), **self.kwargs_dict['plot_params'])

        plt.show()

kwas = KWAs('LSQ')
N = 20
colors = np.random.rand(N)
area = (20 * np.random.rand(N))**2

kwas.add_kwargs_to_dict('scatter_params', s=area, c=colors, alpha=0.5)
kwas.add_kwargs_to_dict('plot_params', linewidth=2.0, color='r')
kwas.list_kwargs()
kwas.get_kwarg_group('scatter_params')
kwas.get_kwarg_group('plot_params')

x = []; y = []
for i in range(N):
    x.append(float(i)*np.pi/float(N))
    y.append(np.sin(x[-1]))

kwas.smoothy(x, y)

I didn't know what parameters you were trying to control with your kwargs, so I made some up from matplotlib examples. The above approach works, and you can add limitless numbers of kwarg groups to the class's kwargs dictionary and add additional methods that can all use the kwargs as desired.

Here's the output using the parameters that I added:

Vandervelde

Answer 4

With a slightly different approach you can use partial. Instead of parameterising your function on the parameters that'll be passed to other functions (like scatter and plot), parameterise on partial signatures of those other functions itself.

def scatter(a, b, z = 345): ...
def plot(d, c, z = 123): ...

def smoothy(x, y, scatter_fn = scatter, plot_fn = plot):
    a = f_a(x, y)
    b = f_b(x, y)
    # Use keyword args yourself so that it quickly errors out if the caller
    # provides those which you intend on explicitly providing.
    scatter_fn(a = x, b = y)
    plot_fn(c = a, d = b)

Callers who are happy with the defaults don't do much:

smoothy(111, 222)

Those who aren't can do:

from functools import partial

smoothy(111, 222, partial(scatter, z = 990), partial(plot, z = 991))

A maybe another advantage is that it allows for better testability since now you can dependency inject the mocked versions of the scatter and/or plot functions. Also, in any-case the user had to know about the usage of scatter and plot inside smoothy because you are advertising the fact that kwargs belong to those functions, so exposing the functions as parameters themselves might not be a disadvantage.

Roemer