Does Python evaluate type hinting of a forward reference?

Question

I was looking at the PEP 484 section on Forward References and noticed the statement:

...that definition may be expressed as a string literal, to be resolved later.

And that got me wondering, when is "later" and by what? The interpreter doesn't try to resolve it as a literal later, so what does? Is it just if a third party tool is written to do that?

Small example to demonstrate the interpreter result:

class A:
    def test(self, a: 'A') -> None:
        pass
class B:
    def test(self, a: A) -> None:
        pass

>>> A().test.__annotations__
{'a': 'A', 'return': None}
>>> B().test.__annotations__
{'a': <class '__main__.A'>, 'return': None}

If my understanding of function annotations and type hints is correct, Python doesn't really do anything with them at runtime to improve performance, but rather the introspective use allows strictly third party applications such as linters, IDEs and static analysis tools (such as mypy) to take advantage of their availability. So would those tools try to resolve the type hint of 'A' rather than having that be a job given to the interpreter and if so, how do they accomplish this?

By using the typing module, user code can perform the following:

>>> typing.get_type_hints(A().test)
{'a': <class '__main__.A'>, 'return': <class 'NoneType'>}
>>> typing.get_type_hints(B().test)
{'a': <class '__main__.A'>, 'return': <class 'NoneType'>}

However, my question is aimed at whether or not Python has any responsibility in updating the __annotations__ of a function from a string literal, that is to say at runtime change:

>>> A().test.__annotations__
{'a': 'A', 'return': None}

to...

>>> A().test.__annotations__
{'a': <class '__main__.A'>, 'return': None}

If Python doesn't do it, then why would I want a string literal as a type hint other than for self-documented code? What value does the first form give to me, a user or a third party tool?

Caducity

mypy doesn't even run your code at all. There's no way it could delegate type hint resolution to Python runtime mechanisms, because runtime doesn't happen. This is true whether or not you use string annotations; it has to do its own resolution no matter what. – Ilia

@Ilia So in what way does using a string literal (simply to avoid a forward reference issue at definition) help? Is it simply self-documented code? A way to say "I would've hinted here, but I had a forward reference so I used a string instead"? Why would ever not use a string literal if thats the case? As in, what purpose does hinting with an actual class add benefit over just a string? Only way I see it is if 3rd party code attempts to evaluate the string. – Caducity

@Ilia There are obviously improvements to this issue since 3.7, I am however trying to understand when forced to use a string literal instead of a class name, what do I lose/gain from that? – Caducity

Answer 1

Consider the following code:

class Foo:
    def bar(self) -> Foo:
        return Foo()

This program will actually crash at runtime if you try running it with Python: when the interpreter sees the definition of bar, the definition of Foo is not yet finished. So, since Foo has not yet been added to the global namespace, we can't use it as a type hint yet.

Similarly, consider this program:

class Foo:
    def bar(self) -> Bar:
        return Bar()

class Bar:
    def foo(self) -> Foo:
        return Foo()

This mutually dependent definition suffers from the same problem: while we're evaluating Foo, Bar hasn't been evaluated yet so the interpreter throws an exception.

---

There are three solutions to this problem. The first is to make some of your type hints strings, effectively "forward declaring" them:

class Foo:
    def bar(self) -> "Foo":
        return Foo()

This satisfies the Python interpreter, and won't disrupt third party tools like mypy: they can just remove the quotes before parsing the type. The main disadvantage is that this syntax looks sort of ugly and clunky.

The second solution is to use type comments syntax:

class Foo:
    def bar(self):
        # type: () -> Foo
        return Foo()

This has the same benefits and disadvantages as the first solution: it satisfies the interpreter and tooling, but looks hacky and ugly. It also has the additional benefit that it keeps your code backwards-compatibile with Python 2.7.

The third solution is Python 3.7+ only -- use the from __future__ import annotations directive:

from __future__ import annotations 

class Foo:
    def bar(self) -> Foo:
        return Foo()

This will automatically make all annotations be represented as strings. So we get the benefit of the first solution, but without the ugliness.

This behavior will eventually become the default in future versions of Python.

It also turns out that automatically making all annotations strings can come with some performance improvements. Constructing types like List[Dict[str, int]] can be surprisingly expensive: they're just regular expressions at runtime and evaluated as if they were written as List.__getitem__(Dict.__getitem__((str, int)).

Evaluating this expression is somewhat expensive: we end up performing two method calls, constructing a tuple, and constructing two objects. This isn't counting any additional work that happens in the __getitem__ methods themselves, of course -- and the work that happens in those methods ends up being non-trivial out of necessity.

(In short, they need to construct special objects that ensure types like List[int] can't be used in inappropriate ways at runtime -- e.g. in isinstance checks and the like.)

Cleanup

Nice breakout, thanks! I do agree with the clunky look of the string literal (which the PEP even says isnt elegant). Using the # type: ... syntax may be a nice since the library I am working with should interface nicely with older S/W. – Caducity

The only piece of type hinting I'm trying to wrap my head around is purpose. The string literal vs class name forward reference is a great example. Since Python doesnt care what its stored as and doesnt have any responsibility in resolving it (as shown in my update), then clearly the main goal of type hints are for 3rd party tools, like mypy or other static analyzers. Is that correct? – Caducity

@pasta_sauce -- yes, type hints are meant almost exclusively for use by third party tools. There are some libraries (e.g. enforce, typeguard) that do try and use type hints at runtime, but they're not very commonly used. – Cleanup

Just came across the third solution in PEP 563; it saved my day! – Hyunhz

For those wondering when the behavior will become a default, the top answer in ask.xiaolee.net/questions/1033177 says it will become default in python 4.0 – Bookstand

As of October 2021, annotations is scheduled to be the default in Python 3.11 (with a planned release date of 2022-10-03). – Parquetry

I came here because I was getting a circular import error because of my type hints, and my thought was to use strings as type hints to avoid the problem but wasn't sure if it was pythonic. I'm happy to see it's going to be the official implementation moving forward! – Izzy

Small but important quibble - the program doesn't "crash", it raises a NameError. Occasionally Python programs do actually crash Python - in other words, the Python interpreter crashes abruptly with a SEGFAULT or SEGV - in my experience, always because of a bad C extension. – Technique