How do I type hint a method with the type of the enclosing class?
Asked Answered
D

9

1268

I have the following code in Python 3:

class Position:

    def __init__(self, x: int, y: int):
        self.x = x
        self.y = y

    def __add__(self, other: Position) -> Position:
        return Position(self.x + other.x, self.y + other.y)

But my editor (PyCharm) says that the reference Position can not be resolved (in the __add__ method). How should I specify that I expect the return type to be of type Position?

I think this is actually a PyCharm issue. It actually uses the information in its warnings, and code completion.

But correct me if I'm wrong, and need to use some other syntax.

Degrease answered 4/11, 2015 at 22:17 Comment(1)
The current state of typing PEPs: mail.python.org/archives/list/[email protected]/message/…Llewellyn
N
1644

TL;DR: As of today (2019), in Python 3.7+ you can turn this feature on using a "future" statement, from __future__ import annotations.

(The behaviour enabled by from __future__ import annotations might become the default in future versions of Python, and was going to be made the default in Python 3.10. However, the change in 3.10 was reverted at the last minute, and now may not happen at all. Python 3.11 does not include it)

In Python 3.6 or below, you should use a string.


I guess you got this exception:

NameError: name 'Position' is not defined

This is because Position must be defined before you can use it in an annotation, unless you are using Python with PEP 563 changes enabled.

Python 3.7+: from __future__ import annotations

Python 3.7 introduces PEP 563: postponed evaluation of annotations. A module that uses the future statement from __future__ import annotations will store annotations as strings automatically:

from __future__ import annotations

class Position:
    def __add__(self, other: Position) -> Position:
        ...

This had been scheduled to become the default in Python 3.10, but this change has now been postponed. Since Python still is a dynamically typed language so no type-checking is done at runtime, typing annotations should have no performance impact, right? Wrong! Before Python 3.7, the typing module used to be one of the slowest python modules in core so for code that involves importing the typing module, you will see an up to 7 times increase in performance when you upgrade to 3.7.

Python <3.7: use a string

According to PEP 484, you should use a string instead of the class itself:

class Position:
    ...
    def __add__(self, other: 'Position') -> 'Position':
       ...

If you use the Django framework, this may be familiar, as Django models also use strings for forward references (foreign key definitions where the foreign model is self or is not declared yet). This should work with Pycharm and other tools.

Sources

The relevant parts of PEP 484 and PEP 563, to spare you the trip:

Forward references

When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.

A situation where this occurs commonly is the definition of a container class, where the class being defined occurs in the signature of some of the methods. For example, the following code (the start of a simple binary tree implementation) does not work:

class Tree:
    def __init__(self, left: Tree, right: Tree):
        self.left = left
        self.right = right

To address this, we write:

class Tree:
    def __init__(self, left: 'Tree', right: 'Tree'):
        self.left = left
        self.right = right

The string literal should contain a valid Python expression (i.e., compile(lit, '', 'eval') should be a valid code object) and it should evaluate without errors once the module has been fully loaded. The local and global namespace in which it is evaluated should be the same namespaces in which default arguments to the same function would be evaluated.

and PEP 563:

Implementation

In Python 3.10, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective __annotations__ dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation.

...

Enabling the future behavior in Python 3.7

The functionality described above can be enabled starting from Python 3.7 using the following special import:

from __future__ import annotations

Things that you may be tempted to do instead

A. Define a dummy Position

Before the class definition, place a dummy definition:

class Position(object):
    pass


class Position(object):
    ...

This will get rid of the NameError and may even look OK:

>>> Position.__add__.__annotations__
{'other': __main__.Position, 'return': __main__.Position}

But is it?

>>> for k, v in Position.__add__.__annotations__.items():
...     print(k, 'is Position:', v is Position)                                                                                                                                                                                                                  
return is Position: False
other is Position: False

B. Monkey-patch in order to add the annotations:

You may want to try some Python metaprogramming magic and write a decorator to monkey-patch the class definition in order to add annotations:

class Position:
    ...
    def __add__(self, other):
        return self.__class__(self.x + other.x, self.y + other.y)

The decorator should be responsible for the equivalent of this:

Position.__add__.__annotations__['return'] = Position
Position.__add__.__annotations__['other'] = Position

At least it seems right:

>>> for k, v in Position.__add__.__annotations__.items():
...     print(k, 'is Position:', v is Position)                                                                                                                                                                                                                  
return is Position: True
other is Position: True

Probably too much trouble.

Normative answered 4/11, 2015 at 22:44 Comment(17)
Right, this is less a PyCharm issue and more a Python 3.5 PEP 484 issue. I suspect you'd get the same warning if you ran it through the mypy type tool.Disgruntle
@JoelBerkeley I just tested it and type parameters worked for me on 3.6, just don't forget to import from typing as any type you use must be in scope when the string is evaluated.Normative
ah, my mistake, i was only putting '' round the class, not the type parametersLennielenno
Important note to anyone using from __future__ import annotations - this must be imported before all other imports.Abdella
This answer fixes the NameError, but produces bad behaviour. The code in the question (and therefore the answer) always returns an instance of Position, when it should probably return an instance of the current class (which might be a subclass). See the answer from @vbraun for type annotations when implementing this correctly.Vincenza
According to python.org/dev/peps/pep-0563/#id5 the new annotation behaviour activates in Python 3.10, not Python 4.Elgin
none of those solutions works if the type is used in a decorator argument before the function declaration, i had to use a string and change the behavior of my personal overload decorator implementation.Devanagari
You said that typing is one of the slowest core modules. Is it possible to use any __future__ module (now, in Python 3.7+) to replace typing's "subscriptable" List, Set etc.?Aniconic
@minseong: __future__ features aren't "modules", and they're only used to enable changes in syntax that can't be fixed by modules. You seem to have missed that typing is already much faster in 3.7, no need to do anything to get the speedup aside from "use 3.7 or higher". If you want to avoid the typing subscriptable types like List entirely, you'll need to go to 3.9, where the standard types (e.g. list) can be subscripted for annotations (e.g. x: list[int]) w/o importing typing.Extemporary
The answer from @vbraun solves the problem of inheritance, but it does this be issuing a new instance of the current class. This answer solves the inheritance problem dynamically, returning the class of the current instance -- not a new instance.Irmgard
Is there a way to specify that the return type of a function is the current class, whatever that may be? e.g., @classmethod def f(cls) -> CurrentClass: where CurrentClass evaluates to whatever cls would be at runtime? So that if A and B inherit from the class that implements f, then A.f() -> A and B.f() -> B?Shindig
@PauloScardine may I suggest adding a shoutout to https://mcmap.net/q/45437/-how-do-i-type-hint-a-method-with-the-type-of-the-enclosing-class as a more inheritance-friendly route? I almost missed that one because it has the least votes so I feel like adding something about it to the body of your (deservedly) high-ranked answer could help people avoid a potential gotcha down the road.Fife
From PEP673: from typing import Self might make this much easier in the future (seems to be PY3.11 feature)Hendren
What about using a string as a dummy definition rather than a class: Position = 'Position'? Would that avoid the identity problem?Corsair
The __future__ import still needs to be used in Python 3.10 as far as I can tell.Prenatal
Python 3.11 introduced the Self annotation. docs.python.org/3.11/whatsnew/3.11.html#whatsnew311-pep673Insufficiency
Thank you very, very much for this information. Not only about what works and why, but also the fact that it was going to be part of Python 3.10, but was dropped just before the release was made.Huttan
G
211

PEP 673 which is implemented in Python 3.11, adds the Self type.

from typing import Self    

class Position:

    def __init__(self, x: int, y: int):
        self.x = x
        self.y = y

    def __add__(self, other: Self) -> Self:
        return type(self)(self.x + other.x, self.y + other.y)

Returning Self is often a good idea, but you must return an object of the same type as self, which means calling type(self) rather than Position.


For older versions of Python (currently 3.7 and later), use the typing-extensions package. One of its purposes is to

Enable use of new type system features on older Python versions. For example, typing.TypeGuard is new in Python 3.10, but typing_extensions allows users on previous Python versions to use it too.

Then you just import from typing_extensions instead of typing, e.g. from typing_extensions import Self.

Gregoor answered 4/11, 2015 at 22:18 Comment(12)
With Python 3.11, this solution become the least kludgy and most succinct.Divergency
Any chance they are back porting this to __future__, etc?Chive
No. __future__ is more about making breaking syntactic features opt-in now, then making it required in a future version. (Which is not to say that a third-party library couldn't provide it now, but it won't be part of the standard library in already existing Python versions.)Gregoor
I believe it's already available as part of typing_extensions, but mypy doesn't understand it yet. The Python 3.11 tracking issue is available here: github.com/python/mypy/issues/12840#issue-1244203018Sere
@Sere Good point, I forgot to check that module.Gregoor
Note this is different from using from __future__ import annotations and annotating with Position, where __add__ on a subclass SubPosition accepts and returns a Position. With Self, it requires and returns a SubPosition. Both approaches can be correct, it depends on the specific use caseLennielenno
(they are of course the same if the class is marked @final)Lennielenno
Self from typing_extensions raises the following error when running mypy on the code: error: Variable "typing_extensions.Self" is not valid as a type and note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliasesEmphasis
As presented, the use of Self as type hint is misleading, since __add__() will always return an instance of Position. Self would be the correct type hint, if in the body of __add__() you'd replace Position with type(self), so that the method returns an instance of the respective subclass. As the code stands now, Self should not be used, but Position with __future__.annotations or 'Position' respectively.Mulch
Ah, good point. Now I'm trying to decide if there are any significant drawbacks to using type(self) in place of Position in the definition. Probably not?Gregoor
Thanks for pointing this out. I just upgraded to Python 3.11 so I could use Self instead of the kludgy TypeVar approach.Huttan
@Sere PEP 673 (Self type) entry in the mypy [github.com/python/mypy/issues/12840#issue-1244203018](issue) you pointed out is now marked as checked.Indamine
S
59

As of Python 3.11 (released in late 2022), there is available typing.Self designed for this purpose. Check PEP 673!

For previous Python versions, one had to consider that the name 'Position' is not available at the time the class body itself is parsed. I don't know how you are using the type declarations, but Python's PEP 484 - which is what most mode should use if using these typing hints say that you can simply put the name as a string at this point:

def __add__(self, other: 'Position') -> 'Position':
    return Position(self.x + other.x, self.y + other.y)

Check the PEP 484 section on forward references - tools conforming to that will know to unwrap the class name from there and make use of it. (It is always important to have in mind that the Python language itself does nothing with these annotations. They are usually meant for static-code analysis, or one could have a library/framework for type-checking at runtime - but you have to explicitly set that.)

Update: Also, as of Python 3.7, check out PEP 563. As of Python 3.8, it is possible to write from __future__ import annotations to defer the evaluation of annotations. Forward-referencing classes should work straightforward.

Update 2: As of Python 3.10, PEP 563 is being retought, and it may be that PEP 649 is used in instead - it would simply allow the class name to be used, plain, without any quotes: the pep proposal is that it is resolved in a lazy way.

Update 3: As of Python 3.11, PEPs 563 and 649 to resolve forward references, mentioned above are still contending and it is likely none of them will go forward as it is now.

Sluggish answered 4/11, 2015 at 22:44 Comment(0)
S
44

Specifying the type as string is fine, but always grates me a bit that we are basically circumventing the parser. So you better not misspell any one of these literal strings:

def __add__(self, other: 'Position') -> 'Position':
    return Position(self.x + other.x, self.y + other.y)

A slight variation is to use a bound typevar, at least then you have to write the string only once when declaring the typevar:

from typing import TypeVar

T = TypeVar('T', bound='Position')

class Position:

    def __init__(self, x: int, y: int):
        self.x = x
        self.y = y

    def __add__(self, other: T) -> T:
        return Position(self.x + other.x, self.y + other.y)
Shrader answered 11/2, 2018 at 19:45 Comment(9)
I wish Python had a typing.Self to specify this explicitly.Plasmodium
I came here looking to see if something like your typing.Self existed. Returning a hard coded string fails to return the correct type when leveraging polymorphism. In my case I wanted to implement a deserialize classmethod. I settled on returning a dict (kwargs) and calling some_class(**some_class.deserialize(raw_data)).Litt
The type annotations used here are appropriate when implementing this correctly to use subclasses. However, the implementation returns Position, and not the class, so the example above is technically incorrect. The implementation should replace Position( with something like self.__class__(.Vincenza
Additionally, the annotations say that the return type depends on other, but most probably it actually depends on self. So, you would need to put the annotation on self to describe the correct behaviour (and maybe other should just be Position to show that it's not tied to the return type). This can also be used for cases when you are only working with self. e.g. def __aenter__(self: T) -> T:Vincenza
The current implementation does not return a T (subclass of Position), but a Position (base class). So the type hint -> T is incorrect. You can of course return type(self)(self.x + other.x, self.y + other.y)Subassembly
@ScottP., @AlexanderHuszagh: There is no need for a typing.Self. Use e.g. def __add__(self: T, other: Position) -> T. You comments prompted me to write an answer.Subassembly
Not only is this answer's T suggestion incorrect, the proposed fixes in the comments are incorrect too - they return an object of self's type, but the annotation says to return other's type. There are ways around that, but this fundamentally doesn't seem like a case where the code should even try for that kind of implicit return type adjustment. Just use 'Position'.Coastguardsman
If you really hate writing out the type name that much, you'll be able to write T: TypeAlias = 'Position' and then use T as an annotation some time soon. TypeAlias recently got added to typing_extensions, and it should be in typing eventually.Coastguardsman
typing.Self will be available in Python 3.11 (according to PEP-673).Gregoor
S
31

If you only care about fixing the NameError: name 'Position' is not defined, you can either specify the class name as a string:

def __add__(self, other: 'Position') -> 'Position':

Or if you use Python 3.7 or higher, add the following line to the top of your code (just before the other imports)

from __future__ import annotations

However, if you also want this to work for subclasses, and return the specific subclass, you need to annotate the method as being a generic method, by using a TypeVar.

What is slightly uncommon is that the TypeVar is bound to the type of self. Basically, this typing hinting tells the type checker that the return type of __add__() and copy() are the same type as self.

from __future__ import annotations

from typing import TypeVar

T = TypeVar('T', bound=Position)

class Position:
    
    def __init__(self, x: int, y: int):
        self.x = x
        self.y = y
    
    def __add__(self: T, other: Position) -> T:
        return type(self)(self.x + other.x, self.y + other.y)
    
    def copy(self: T) -> T:
        return type(self)(self.x, self.y)
Subassembly answered 3/8, 2020 at 21:32 Comment(6)
@Arjan. you are right. I'm so accustomed to from __future__ import annotations that I probably forgot. Thanks for pointing this out. I fixed it in the answer.Subassembly
what is the letter ' T ' ?Veronikaveronike
@Eildosa: "T" is defined as a TypeVar. Think of it as "any type". In the defintion copy(self: T) -> T this means that whatever object you throw at copy(), copy() will always return an object of the same type. In this case, T is a TypeVar "bound" to Postion, which means "any type that is either Position, or a subclass of Position". Search for TypeVar to learn more about it.Subassembly
Are there any clever tricks to have a generic Self that can be reused?Xerosis
How does that look for a @classmethod?Alerion
Isn't T = TypeVar('T', bound=Position) referencing Position before it is defined?Myxomatosis
W
18

When a string-based type hint is acceptable, the __qualname__ item can also be used. It holds the name of the class, and it is available in the body of the class definition.

class MyClass:
    @classmethod
    def make_new(cls) -> __qualname__:
        return cls()

By doing this, renaming the class does not imply modifying the type hints. But I personally would not expect smart code editors to handle this form well.

Winnipegosis answered 10/11, 2019 at 13:59 Comment(6)
This is especially useful because it does not hardcode the class name, so it keeps working in subclasses.Warbeck
I'm not sure whether this will work with the postponed evaluation of annotations (PEP 563), so I've asked a question for that.Warbeck
Note that this is not a valid annotation as far as mypy is concerned.Cavity
this solution fixes the hardcoding in a different mannerIrmgard
@user2426679 both this answer and the one you are referencing are not valid type annotations. Use the bound typevar approach here: https://mcmap.net/q/45437/-how-do-i-type-hint-a-method-with-the-type-of-the-enclosing-classStefa
This works well in IntelliJ IDECordilleras
I
8

edit: @juanpa.arrivillaga brought to my attention a better way to do this; see https://stackoverflow.com/a/63237226

It's recommended to do the above answer instead of this one below.

[old answer below, kept for posterity]

I ❤️ Paulo's answer

However, there's a point to be made about type hint inheritance in relation to self, which is that if you type hint by using a literal copy paste of the class name as a string, then your type hint won't inherit in a correct or consistent way.

The solution to this is to provide return type hint by putting the type hint on the return in the function itself.

✅ For example, do this:

class DynamicParent:
  def func(self):
    # roundabout way of returning self in order to have inherited type hints of the return
    # https://stackoverflow.com/a/64938978
    _self:self.__class__ = self
    return _self

Instead of doing this:

class StaticParent:
  def func(self) -> 'StaticParent':
    return self

Below is the reason why you want to do the type hint via the roundabout ✅ way shown above

class StaticChild(StaticParent):
  pass

class DynamicChild(DynamicParent):
  pass

static_child = StaticChild()
dynamic_child = DynamicChild()

dynamic_child screenshot shows that type hinting works correctly when referencing the self:

enter image description here

static_child screenshot shows that type hinting is mistakenly pointing at the parent class, i.e. the type hint does not change correctly with inheritance; it is static because it will always point at the parent even when it should point at the child

enter image description here

Irmgard answered 21/11, 2020 at 1:33 Comment(4)
this is not a valid type annotation, and not the correct way to type annotate what you are trying to express, which should be annotated with a type variable bound to the parent classStefa
@Stefa could you post an answer to this question that is annotated with a type variable bound to the parent class? It's unclear to me how one would bind a type variable to the parent class that refers to the subsequent children instances.Irmgard
See: https://mcmap.net/q/45437/-how-do-i-type-hint-a-method-with-the-type-of-the-enclosing-classStefa
Nice! It works with VSCode Intellisense. I am wondering if this assignment _self:self.__class__ = self would introduce any overhead (?)Wore
F
1

For Python 3.11+ there is a 'Self' type hint in the 'typings' module.

For Cython/mypyc users who might have stumbled into this like me; it doesn't matter.. The compiler is smart enough to infer that the type specified in the function parameter correspond to the enclosing class type

Fosdick answered 13/8, 2023 at 3:16 Comment(0)
V
0

Use:

from __future__ import annotations

import sys

if sys.version_info >= (3, 11):
    from typing import Self
else:
    from typing_extensions import Self


class Animal:
    def __init__(self, name: str, says: str) -> None:
        self.name = name
        self.says = says

    @classmethod
    def from_description(cls, description: str = "|") -> Self:
        descr = description.split("|")
        return cls(descr[0], descr[1])
Vocative answered 10/3, 2023 at 1:37 Comment(3)
The link is (effectively) broken. It redirects to a generic page.Fidole
An explanation would be in order. E.g., what is the idea/gist? From the Help Center: "...always explain why the solution you're presenting is appropriate and how it works". Please respond by editing (changing) your answer, not here in comments (but *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** without *** *** *** *** *** *** *** *** *** *** "Edit:", "Update:", or similar - the answer should appear as if it was written today).Fidole
Related: What do we do with answers that are entirely copied and improperly attributed (only a "reference" link or similar is included)?Fidole

© 2022 - 2024 — McMap. All rights reserved.