Python 3 type hinting for decorator

Question

Consider the following code:

from typing import Callable, Any

TFunc = Callable[..., Any]

def get_authenticated_user(): return "John"

def require_auth() -> Callable[TFunc, TFunc]:
    def decorator(func: TFunc) -> TFunc:
        def wrapper(*args, **kwargs) -> Any:
            user = get_authenticated_user()
            if user is None:
                raise Exception("Don't!")
            return func(*args, **kwargs)
        return wrapper
    return decorator

@require_auth()
def foo(a: int) -> bool:
    return bool(a % 2)

foo(2)      # Type check OK
foo("no!")  # Type check failing as intended

This piece of code is working as intended. Now imagine I want to extend this, and instead of just executing func(*args, **kwargs) I want to inject the username in the arguments. Therefore, I modify the function signature.

from typing import Callable, Any

TFunc = Callable[..., Any]

def get_authenticated_user(): return "John"

def inject_user() -> Callable[TFunc, TFunc]:
    def decorator(func: TFunc) -> TFunc:
        def wrapper(*args, **kwargs) -> Any:
            user = get_authenticated_user()
            if user is None:
                raise Exception("Don't!")
            return func(*args, user, **kwargs)  # <- call signature modified

        return wrapper

    return decorator


@inject_user()
def foo(a: int, username: str) -> bool:
    print(username)
    return bool(a % 2)


foo(2)      # Type check OK
foo("no!")  # Type check OK <---- UNEXPECTED

I can't figure out a correct way to type this. I know that on this example, decorated function and returned function should technically have the same signature (but even that is not detected).

Estranged

Callable arguments are really hard to type; various proposals are floating around but currently, the only thing I can tell you is to continue to use .... – Roanna

That's what I thought too. I did not find anything, except some issues opened on GitHub suggesting adding typing such as StarArg and StarKwarg. I wonder if there was a solution in this very particular case, but I think there won't be any :( – Estranged

You found the discussions the type hinting developers are conducting to improve this situation. – Roanna

Answer 1

You can't use Callable to say anything about additional arguments; they are not generic. Your only option is to say that your decorator takes a Callable and that a different Callable is returned.

In your case you can nail down the return type with a typevar:

RT = TypeVar('RT')  # return type

def inject_user() -> Callable[[Callable[..., RT]], Callable[..., RT]]:
    def decorator(func: Callable[..., RT]) -> Callable[..., RT]:
        def wrapper(*args, **kwargs) -> RT:
            # ...

Even then the resulting decorated foo() function has a typing signature of def (*Any, **Any) -> builtins.bool* when you use reveal_type().

Various proposals are currently being discussed to make Callable more flexible but those have not yet come to fruition. See

• Allow variadic generics
• Proposal: Generalize Callable to be able to specify argument names and kinds
• TypeVar to represent a Callable's arguments
• Support function decorators excellently

for some examples. The last one in that list is an umbrella ticket that includes your specific usecase, the decorator that alters the callable signature:

Mess with the return type or with arguments

For an arbitrary function you can't do this at all yet -- there isn't even a syntax. Here's me making up some syntax for it.

Roanna

Answer 2

PEP 612 was accepted after the accepted answer, and we now have typing.ParamSpec and typing.Concatenate in Python 3.10. With these variables, we can correctly type some decorators that manipulate positional parameters.

The code in question can be typed like this:

from typing import Callable, ParamSpec, Concatenate, TypeVar

Param = ParamSpec("Param")
RetType = TypeVar("RetType")

def get_authenticated_user()->str:
    return "John"

def inject_user() -> Callable[[Callable[Param, RetType]], Callable[Concatenate[str, Param], RetType]]:
    def decorator(func: Callable[Param, RetType]) -> Callable[Concatenate[str, Param], RetType]:
        def wrapper(user: str, *args:Param.args, **kwargs:Param.kwargs) -> RetType:
            user = get_authenticated_user()
            if user is None:
                raise Exception("Don't!")
            return func(*args, **kwargs)

        return wrapper

    return decorator


@inject_user()
def foo(a: int) -> bool:
    return bool(a % 2)


reveal_type(foo)  #  # I: Revealed type is "def (builtins.str, a: builtins.int) -> builtins.bool"

foo("user", 2)  # Type check OK
foo("no!")  # E: Missing positional argument "a" in call to "foo"  [call-arg]
foo(3)  # # E: Missing positional argument "a" in call to "foo"  [call-arg] # E: Argument 1 to "foo" has incompatible type "int"; expected "str"  [arg-type]

Hammerhead

Now that support for this has landed in mypy & Python 3.10 I tried this out, but I get this error on the typing of the Callable declarations (Param = ParamSpec("Param"); MyFunc = Callable[Param, RetType]): mypy: error: The first argument to Callable must be a list of types, parameter specification, or "..." – Cadet

There is a mistake in Concatenate, it should be Concatenate[str, Param]. ParamSpec has to come last. – Kellsie

The tracking mypy issue for PEP 612 support is now fixed and closed. – Revegetate

The wrapper function arguments still needed typing (in my case, at least) def wrapper(*args: Param.args, **kwargs: Param.kwargs) -> RetType: – Verdin

Hmm… I can’t make the above work, with or without strict mode 🤔 That’s mypy v1.7.1 with only default settings. – Legg

This works now, but it's tricky business. Mainly, it seems you can't use a TypeAlias (like OriginalFunc/DecoratedFunc) to reference a TypeVar, or mypy will not know what RetType refers to. As far as I've been able to tell, you have to write out the Callables literally. – Extracurricular

Answer 3

I tested this in Pyright.

from typing import Any, Callable, Type, TypeVar

T = TypeVar('T')

def typing_decorator(rtype: Type[T]) -> Callable[..., Callable[..., T]]:
    """
    Useful function to typing a previously decorated func.
    ```
    @typing_decorator(rtype = int)
    @my_decorator()
    def my_func(a, b, *c, **d):
        ...
    ```
    In Pyright the return typing of my_func will be int.
    """
    def decorator(function: Any) -> Any:
        def wrapper(*args: Any, **kwargs: Any) -> Any:
            return function(*args, **kwargs)
        return wrapper
    return decorator  # type: ignore

Foregoing

Answer 4

The problem is solved using the decohints library:

pip install decohints

Here is how it will work with your code:

from decohints import decohints


def get_authenticated_user():
    return "John"


@decohints
def inject_user():
    def decorator(func):
        def wrapper(*args, **kwargs):
            user = get_authenticated_user()
            if user is None:
                raise Exception("Don't!")
            return func(*args, user, **kwargs)  # <- call signature modified

        return wrapper

    return decorator


@inject_user()
def foo(a: int, username: str) -> bool:
    print(username)
    return bool(a % 2)

If you type below foo() in PyCharm and wait, it will show foo function parameter hints (a: int, username: str).

Here is a link to the decohints sources, there are also other options for solving this problem: https://github.com/gri-gus/decohints

Pippin

To save people a click, here's the entire implementation of decohints: def decohints(decorator: Callable) -> Callable: return decorator – Puca

@DavidRöthlisberger LOL – Kellsie