Set function signature in Python

Question

Suppose I have a generic function f. I want to programmatically create a function f2 that behaves the same as f, but has a customized signature.

More detail

Given a list l and and dictionary d I want to be able to:

• Set the non-keyword arguments of f2 to the strings in l
• Set the keyword arguments of f2 to the keys in d and the default values to the values of d

ie. Suppose we have

l = ["x", "y"]
d = {"opt": None}

def f(*args, **kwargs):
    # My code

Then I would want a function with signature:

def f2(x, y, opt=None):
    # My code

A specific use case

This is just a simplified version of my specific use case. I am giving this as an example only.

My actual use case (simplified) is as follows. We have a generic initiation function:

def generic_init(self, *args, **kwargs):
    """Function to initiate a generic object"""
    for name, arg in zip(self.__init_args__, args):
        setattr(self, name, arg)
    for name, default in self.__init_kw_args__.items():
        if name in kwargs:
            setattr(self, name, kwargs[name])
        else:
            setattr(self, name, default)

We want to use this function in a number of classes. In particular, we want to create a function __init__ that behaves like generic_init, but has the signature defined by some class variables at creation time:

class my_class:
    __init_args__ = ["x", "y"]
    __kw_init_args__ = {"my_opt": None}

__init__ = create_initiation_function(my_class, generic_init)
setattr(myclass, "__init__", __init__)

We want create_initiation_function to create a new function with the signature defined using __init_args__ and __kw_init_args__. Is it possible to write create_initiation_function?

Please note:

• If I just wanted to improve the help, I could set __doc__.
• We want to set the function signature on creation. After that, it doesn't need to be changed.
• Instead of creating a function like generic_init, but with a different signature we could create a new function with the desired signature that just calls generic_init
• We want to define create_initiation_function. We don't want to manually specify the new function!

Related

• Preserving signatures of decorated functions: This is how to preserve a signature when decorating a function. We need to be able to set the signature to an arbitrary value

Kuvasz

Are you asking how to programaticaly create such a function? Anyway, i think you need to give us some context. How do you want to use it? How do you want to specify body of the function? Could you show us how you would like to use it? – Mcgurn

Is it clearer now that I've edited it again? – Kuvasz

@tomekszpakowicz That's exactly it. I want to be able to programmaticlly create the body. – Kuvasz

Ah, do you want to change the signature, without actually changing the function!? You can't do that. At least, you should not be able to do that, for a purely moral standpoint. Again I have to ask you what your actual problem is. What are you trying to accomplish? It seems to me highly unlikely that what ever you are trying to do is actually the best solution to the problem. – Lastex

@Lennart. Copying a function and changing the signature would be a solution if it were possible. I think that it is much more feasible to create a new function with a different signature that calls the generic function. Not quite as neat though. – Kuvasz

And so the problem description grows and grows... – Kuvasz

@Casebash: I think that is a very neat feature. You have a generic function, and you want to create several non-generic functions with a specific signature. The neatest way of doing that is to just create them. – Lastex

@Kuvasz You got a lot of ridicule for this but I found this to be a reasonable idea. Not feasible, but the idea is very cool. – Benefice

Everyone who criticized this question needs to go study namedtuple right now. It does exactly the task which the OP requested information on how to accomplish (programmatically creates a function signature). – Spurn

Answer 1

For your usecase, having a docstring in the class/function should work -- that will show up in help() okay, and can be set programmatically (func.__doc__ = "stuff").

I can't see any way of setting the actual signature. I would have thought the functools module would have done it if it was doable, but it doesn't, at least in py2.5 and py2.6.

You can also raise a TypeError exception if you get bad input.

Hmm, if you don't mind being truly vile, you can use compile()/eval() to do it. If your desired signature is specified by arglist=["foo","bar","baz"], and your actual function is f(*args, **kwargs), you can manage:

argstr = ", ".join(arglist)
fakefunc = "def func(%s):\n    return real_func(%s)\n" % (argstr, argstr)
fakefunc_code = compile(fakefunc, "fakesource", "exec")
fakeglobals = {}
eval(fakefunc_code, {"real_func": f}, fakeglobals)
f_with_good_sig = fakeglobals["func"]

help(f)               # f(*args, **kwargs)
help(f_with_good_sig) # func(foo, bar, baz)

Changing the docstring and func_name should get you a complete solution. But, uh, eww...

Steinway

I should have thought of eval. Anything is possible with it. – Kuvasz

This is the pattern used by the decorator package -- pypi.python.org/pypi/decorator – Wilkie

completely vile? this is exactly what namedtuple does. – Spurn

You should not use eval unless you can help it. Consider using the python-forge package I suggest in my answer: https://mcmap.net/q/346373/-set-function-signature-in-python – Tedman

Answer 2

From PEP-0362, there actually does appear to be a way to set the signature in py3.3+, using the fn.__signature__ attribute:

from inspect import signature
from functools import wraps

def shared_vars(*shared_args):
    """Decorator factory that defines shared variables that are
       passed to every invocation of the function"""

    def decorator(f):
        @wraps(f)
        def wrapper(*args, **kwargs):
            full_args = shared_args + args
            return f(*full_args, **kwargs)

        # Override signature
        sig = signature(f)
        sig = sig.replace(parameters=tuple(sig.parameters.values())[1:])
        wrapper.__signature__ = sig

        return wrapper
    return decorator

Then:

>>> @shared_vars({"myvar": "myval"})
>>> def example(_state, a, b, c):
>>>     return _state, a, b, c
>>> example(1,2,3)
({'myvar': 'myval'}, 1, 2, 3)
>>> str(signature(example))
'(a, b, c)'

Note: the PEP is not exactly right; Signature.replace moved the params from a positional arg to a kw-only arg.

Soothfast

Consider using the python-forge package I suggest in my answer: https://mcmap.net/q/346373/-set-function-signature-in-python – I designed it to solve this problem. – Tedman

Attempting to make the signature larger, I'm always getting a TypeError due to unexpected Param. I'll probably open a new issue but if anyone has any hints do tell. – Eggert

It is useful indeed. – Kenji

Thanks for your post. It works in my scenario. – Nena

Some type-checkers complain that __signature__ is not assignable. It still works of course as Python allows it. But it makes me wonder if there's a... "cleaner" way of doing it? – Hamachi

Answer 3

I wrote a package named forge that solves this exact problem for Python 3.5+:

With your current code looking like this:

l=["x", "y"]
d={"opt":None}

def f(*args, **kwargs):
    #My code

And your desired code looking like this:

def f2(x, y, opt=None):
    #My code

Here is how you would solve that using forge:

f2 = forge.sign(
    forge.arg('x'),
    forge.arg('y'),
    forge.arg('opt', default=None),
)(f)

As forge.sign is a wrapper, you could also use it directly:

@forge.sign(
    forge.arg('x'),
    forge.arg('y'),
    forge.arg('opt', default=None),
)
def func(*args, **kwargs):
    # signature becomes: func(x, y, opt=None)
    return (args, kwargs)

assert func(1, 2) == ((), {'x': 1, 'y': 2, 'opt': None})

Tedman

Surprise, surprise, the package is no longer maintained, despite the @Devin's heavy advertising a couple of years ago. I pity those who added this dependency to their projects. – Canaille

Answer 4

Have a look at makefun, it was made for that (exposing variants of functions with more or less parameters and accurate signature), and works in python 2 and 3.

Your example would be written like this:

try:  # python 3.3+
    from inspect import signature, Signature, Parameter
except ImportError:
    from funcsigs import signature, Signature, Parameter

from makefun import create_function

def create_initiation_function(cls, gen_init):
    # (1) check which signature we want to create
    params = [Parameter('self', kind=Parameter.POSITIONAL_OR_KEYWORD)]
    for mandatory_arg_name in cls.__init_args__:
        params.append(Parameter(mandatory_arg_name, kind=Parameter.POSITIONAL_OR_KEYWORD))
    for default_arg_name, default_arg_val in cls.__opt_init_args__.items():
        params.append(Parameter(default_arg_name, kind=Parameter.POSITIONAL_OR_KEYWORD, default=default_arg_val))
    sig = Signature(params)

    # (2) create the init function dynamically
    return create_function(sig, generic_init)

# ----- let's use it

def generic_init(self, *args, **kwargs):
    """Function to initiate a generic object"""
    assert len(args) == 0
    for name, val in kwargs.items():
        setattr(self, name, val)

class my_class:
    __init_args__ = ["x", "y"]
    __opt_init_args__ = {"my_opt": None}

my_class.__init__ = create_initiation_function(my_class, generic_init)

and works as expected:

# check 
o1 = my_class(1, 2)
assert vars(o1) == {'y': 2, 'x': 1, 'my_opt': None}

o2 = my_class(1, 2, 3)
assert vars(o2) == {'y': 2, 'x': 1, 'my_opt': 3}

o3 = my_class(my_opt='hello', y=3, x=2)
assert vars(o3) == {'y': 3, 'x': 2, 'my_opt': 'hello'}

Silverman

Just wanted to let you know: Thanks so much for the lib! I was working so hard on getting the signature in my decorator right. Nothing worked including trying to use chriswarrick.com/blog/2018/09/20/… as inspiration. There I saw your comment, got to your lib (and later here). Thanks again! Using your lib was so easy. – Treasonable

Thanks for the kind words @Treasonable ! I very much appreciate it – Silverman

Answer 5

You can't do this with live code.

That is, you seem to be wanting to take an actual, live function that looks like this:

def f(*args, **kwargs):
    print args[0]

and change it to one like this:

def f(a):
    print a

The reason this can't be done--at least without modifying actual Python bytecode--is because these compile differently.

The former results in a function that receives two parameters: a list and a dict, and the code you're writing operates on that list and dict. The second results in a function that receives one parameter, and which is accessed as a local variable directly. If you changed the function "signature", so to speak, it'd result in a function like this:

def f(a):
    print a[0]

which obviously wouldn't work.

If you want more detail (though it doesn't really help you), a function that takes an *args or *kwargs has one or two bits set in f.func_code.co_flags; you can examine this yourself. The function that takes a regular parameter has f.func_code.co_argcount set to 1; the *args version is 0. This is what Python uses to figure out how to set up the function's stack frame when it's called, to check parameters, etc.

If you want to play around with modifying the function directly--if only to convince yourself that it won't work--see this answer for how to create a code object and live function from an existing one to modify bits of it. (This stuff is documented somewhere, but I can't find it; it's nowhere in the types module docs...)

That said, you can dynamically change the docstring of a function. Just assign to func.__doc__. Be sure to only do this at load time (from the global context or--most likely--a decorator); if you do it later on, tools that load the module to examine docstrings will never see it.

Sentience

I don't need to dynamically change the signature of a function, just to be able create a function which does the same as another function and has a different signature – Kuvasz

Not quite. As pointed out by Towns, we could just create a new function using eval that can call the generic function – Kuvasz

Answer 6

Maybe I didn't understand the problem well, but if it's about keeping the same behavior while changing the function signature, then you can do something like :

# define a function
def my_func(name, age) :
    print "I am %s and I am %s" % (name, age)

# label the function with a backup name
save_func = my_func

# rewrite the function with a different signature
def my_func(age, name) :
    # use the backup name to use the old function and keep the old behavior
    save_func(name, age)

# you can use the new signature
my_func(35, "Bob")

This outputs :

I am Bob and I am 35

Inflict

Needs to be programmatically. Its hard to explain, but I suppose one last edit wouldn't kill me – Kuvasz

Answer 7

We want create_initiation_function to change the signature

Please don't do this.

We want to use this function in a number of classes

Please use ordinary inheritance.

There's no value in having the signature "changed" at run time.

You're creating a maintenance nightmare. No one else will ever bother to figure out what you're doing. They'll simply rip it out and replace it with inheritance.

Do this instead. It's simple and obvious and makes your generic init available in all subclasses in an obvious, simple, Pythonic way.

class Super( object ):
    def __init__( self, *args, **kwargs ):
        # the generic __init__ that we want every subclass to use

class SomeSubClass( Super ):
    def __init__( self, this, that, **kwdefaults ):
        super( SomeSubClass, self ).__init__( this, that, **kwdefaults )

class AnotherSubClass( Super ):
    def __init__( self, x, y, **kwdefaults ):
        super( AnotherSubClass, self ).__init__( x, y, **kwdefaults )

Gunshot

Answer 8

Edit 1: Answering new question:

You ask how you can create a function with this signature:

def fun(a, b, opt=None):
    pass

The correct way to do that in Python is thus:

def fun(a, b, opt=None):
    pass

Edit 2: Answering explanation:

"Suppose I have a generic function f. I want to programmatically create a function f2 that behaves the same as f, but has a customised signature."

def f(*args, **kw):
    pass

OK, then f2 looks like so:

def f2(a, b, opt=None):
    f(a, b, opt=opt)

Again, the answer to your question is so trivial, that you obviously want to know something different that what you are asking. You really do need to stop asking abstract questions, and explain your concrete problem.

Lastex

Doesn't answer the question. Sorry for being unclear, I have clarified it now. – Kuvasz

It did answer the question. I've also updated the answer to reflect your changes. It again answers the question. It probably doens't tell you what you want to know, but you have to tell us what you want to know first. – Lastex

I was pretty sure that the meaning was clear after I edited it, but I have edited it again. – Kuvasz

No, it's still not clear. What do you mean by "create a function"? What is that function supposed to do? – Spotty

Okay, sorry for everyone who I have confused. I really hope I've managed to make myself clear this time. – Kuvasz

Programmatically is the key word – Kuvasz

Yeah, but the code is the same no matter of you create that code by typing it in or with a program. – Lastex