Preserving signatures of decorated functions

Question

Suppose I have written a decorator that does something very generic. For example, it might convert all arguments to a specific type, perform logging, implement memoization, etc.

Here is an example:

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

>>> funny_function("3", 4.0, z="5")
22

Everything well so far. There is one problem, however. The decorated function does not retain the documentation of the original function:

>>> help(funny_function)
Help on function g in module __main__:

g(*args, **kwargs)

Fortunately, there is a workaround:

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

This time, the function name and documentation are correct:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(*args, **kwargs)
    Computes x*y + 2*z

But there is still a problem: the function signature is wrong. The information "*args, **kwargs" is next to useless.

What to do? I can think of two simple but flawed workarounds:

1 -- Include the correct signature in the docstring:

def funny_function(x, y, z=3):
    """funny_function(x, y, z=3) -- computes x*y + 2*z"""
    return x*y + 2*z

This is bad because of the duplication. The signature will still not be shown properly in automatically generated documentation. It's easy to update the function and forget about changing the docstring, or to make a typo. [And yes, I'm aware of the fact that the docstring already duplicates the function body. Please ignore this; funny_function is just a random example.]

2 -- Not use a decorator, or use a special-purpose decorator for every specific signature:

def funny_functions_decorator(f):
    def g(x, y, z=3):
        return f(int(x), int(y), z=int(z))
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

This works fine for a set of functions that have identical signature, but it's useless in general. As I said in the beginning, I want to be able to use decorators entirely generically.

I'm looking for a solution that is fully general, and automatic.

So the question is: is there a way to edit the decorated function signature after it has been created?

Otherwise, can I write a decorator that extracts the function signature and uses that information instead of "*kwargs, **kwargs" when constructing the decorated function? How do I extract that information? How should I construct the decorated function -- with exec?

Any other approaches?

Walston

Never said "out of date". I was more or less wondering what inspect.Signature added to dealing with decorated functions. – Gerbold

Answer 1

1. Install decorator module:

   $ pip install decorator
   

2. Adapt definition of args_as_ints():

   import decorator
   
   @decorator.decorator
   def args_as_ints(f, *args, **kwargs):
       args = [int(x) for x in args]
       kwargs = dict((k, int(v)) for k, v in kwargs.items())
       return f(*args, **kwargs)
   
   @args_as_ints
   def funny_function(x, y, z=3):
       """Computes x*y + 2*z"""
       return x*y + 2*z
   
   print funny_function("3", 4.0, z="5")
   # 22
   help(funny_function)
   # Help on function funny_function in module __main__:
   # 
   # funny_function(x, y, z=3)
   #     Computes x*y + 2*z
   

---

Python 3.4+

functools.wraps() from stdlib preserves signatures since Python 3.4:

import functools


def args_as_ints(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)
    return wrapper


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z


print(funny_function("3", 4.0, z="5"))
# 22
help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

functools.wraps() is available at least since Python 2.5 but it does not preserve the signature there:

help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(*args, **kwargs)
#    Computes x*y + 2*z

Notice: *args, **kwargs instead of x, y, z=3.

Stacked

Yours wasn't the first answer, but the most comprehensive so far :-) I would actually prefer a solution not involving a third party module, but looking at the source for the decorator module, it's simple enough that I'll be able to just copy it. – Walston

To preserve the signature, additionally set wrapper.__signature__ = inspect.signature(func) before returning wrapper. – Amenity

@MarkLodato: functools.wraps() already preserves signatures in Python 3.4+ (as said in the answer). Do you mean setting wrapper.__signature__ helps on earlier versions? (which versions have you tested?) – Stacked

Ah, sorry. inspect.signature is not available in Python 2 so it won't help. In Python 3.4, functools.wraps() does not set __signature__ so the signature does not show up properly in IPython. Setting it explicitly fixes the issue. So this doesn't solve the OP's problem, but it did solve mine. – Amenity

@MarkLodato: help() shows the correct signature on Python 3.4. Why do you think functools.wraps() is broken and not IPython? – Stacked

@J.F.Sebastian: functools.wraps() is not broken, but older versions of IPython3 don't pick up the signature unless you set __signature__. For example, the latest version in Ubuntu (1.2.1) requires __signature__ to be set, while it works in 3.2.1 from pip. See gist.github.com/anonymous/3d97523517abe0f55dc9. – Amenity

@MarkLodato: it is broken if we have to write code to fix it. Given that help() produces the correct result, the question is what piece of software should be fixed: functools.wraps() or IPython? In any case, manually assigning __signature__ is a workaround at best -- it is not a long-term solution. – Stacked

Looks like inspect.getfullargspec() still doesn't return proper signature for functools.wraps in python 3.4 and that you must use inspect.signature() instead. – Kofu

It depends on what you mean by "preserve signature": appearance or behavior. As of python 3.7, functools.wraps still does not preserve the behavior (TypeError in case of wrong arguments). See detailed answer here: https://mcmap.net/q/87826/-preserving-signatures-of-decorated-functions – Sonjasonnet

Answer 2

This is solved with Python's standard library functools and specifically functools.wraps function, which is designed to "update a wrapper function to look like the wrapped function". It's behaviour depends on Python version, however, as shown below. Applied to the example from the question, the code would look like:

from functools import wraps

def args_as_ints(f):
    @wraps(f) 
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

When executed in Python 3, this would produce the following:

>>> funny_function("3", 4.0, z="5")
22
>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

Its only drawback is that in Python 2 however, it doesn't update function's argument list. When executed in Python 2, it will produce:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(*args, **kwargs)
    Computes x*y + 2*z

Huberman

Not sure if it's Sphinx, but this doesn't seem to work when the wrapped function is a method of a class. Sphinx continues to report the call signature of the decorator. – Cryan

functools.wraps is incomplete. inspect.getfullargspec(func) still returns the signature of the decorator function instead of the wrapped function. – Judicial

Answer 3

There is a decorator module with decorator decorator you can use:

@decorator
def args_as_ints(f, *args, **kwargs):
    args = [int(x) for x in args]
    kwargs = dict((k, int(v)) for k, v in kwargs.items())
    return f(*args, **kwargs)

Then the signature and help of the method is preserved:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

EDIT: J. F. Sebastian pointed out that I didn't modify args_as_ints function -- it is fixed now.

Monadism

Answer 4

Take a look at the decorator module - specifically the decorator decorator, which solves this problem.

Martyrology

Answer 5

Second option:

1. Install wrapt module:

$ easy_install wrapt

wrapt have a bonus, preserve class signature.

import wrapt
import inspect

@wrapt.decorator
def args_as_ints(wrapped, instance, args, kwargs):
    if instance is None:
        if inspect.isclass(wrapped):
            # Decorator was applied to a class.
            return wrapped(*args, **kwargs)
        else:
            # Decorator was applied to a function or staticmethod.
            return wrapped(*args, **kwargs)
    else:
        if inspect.isclass(instance):
            # Decorator was applied to a classmethod.
            return wrapped(*args, **kwargs)
        else:
            # Decorator was applied to an instancemethod.
            return wrapped(*args, **kwargs)


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x * y + 2 * z


>>> funny_function(3, 4, z=5))
# 22

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

Ability

Answer 6

As commented above in jfs's answer ; if you're concerned with signature in terms of appearance (help, and inspect.signature), then using functools.wraps is perfectly fine.

If you're concerned with signature in terms of behavior (in particular TypeError in case of arguments mismatch), functools.wraps does not preserve it. You should rather use decorator for that, or my generalization of its core engine, named makefun.

from makefun import wraps

def args_as_ints(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("wrapper executes")
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)
    return wrapper


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z


print(funny_function("3", 4.0, z="5"))
# wrapper executes
# 22

help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

funny_function(0)  
# observe: no "wrapper executes" is printed! (with functools it would)
# TypeError: funny_function() takes at least 2 arguments (1 given)

See also this post about functools.wraps.

Sonjasonnet

Also, the result of inspect.getfullargspec is not kept by calling functools.wraps. – Applaud

Thanks for the useful additional comment @Applaud ! – Sonjasonnet

Answer 7

from inspect import signature


def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    sig = signature(f)
    g.__signature__ = sig
    g.__doc__ = f.__doc__
    g.__annotations__ = f.__annotations__
    g.__name__ = f.__name__
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

>>> funny_function("3", 4.0, z="5")
22

I wanted to add that answer (since this shows up first in google). The inspect module is able to fetch the signature of a function, so that it can be preserved in decorators. But that's not all. If you want to modify the signature, you can do so like this :

from inspect import signature, Parameter, _ParameterKind


def foo(a: int, b: int) -> int:
    return a + b

sig = signature(foo)
sig._parameters = dict(sig.parameters)
sig.parameters['c'] = Parameter(
    'c', _ParameterKind.POSITIONAL_OR_KEYWORD, 
    annotation=int
)
foo.__signature__ = sig

>>> help(foo)
Help on function foo in module __main__:

foo(a: int, b: int, c: int) -> int

Why would you want to mutate a function's signature ?

It's mostly useful to have adequate documentation on your functions and methods. If you're using the *args, **kwargs syntax and then popping arguments from kwargs for other uses in your decorators, that keyword argument won't be properly documented, hence, modifying the signature of the function.

Sulfapyrazine

Answer 8

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

this fixes name and documentation. to preserve the function signature, wrap is used exactly at same location as g.__name__ = f.__name__, g.__doc__ = f.__doc__.

the wraps itself a decorator. we pass the closure-the inner function to that decorator, and it is going to fix up the metadata. BUt if we only pass in the inner function to wraps, it is not gonna know where to copy the metadata from. It needs to know which function's metadata needs to be protected. It needs to know the original function.

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g=wraps(f)(g)
    return g

wraps(f) is going to return a function which will take g as its parameter. And that is going to return closure and will assigned to g and then we return it.

Hierarchize