AttributeErrors: undesired interaction between @property and __getattr__

Question

I have a problem with AttributeErrors raised in a @property in combination with __getattr__() in python:

Example code:

>>> def deeply_nested_factory_fn():
...     a = 2
...     return a.invalid_attr
...
>>> class Test(object):
...     def __getattr__(self, name):
...         if name == 'abc':
...             return 'abc'
...         raise AttributeError("'Test' object has no attribute '%s'" % name)
...     @property
...     def my_prop(self):
...         return deeply_nested_factory_fn()
...
>>> test = Test()
>>> test.my_prop
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __getattr__
AttributeError: 'Test' object has no attribute 'my_prop'

In my case, this is a highly misleading error message, because it hides the fact that deeply_nested_factory_fn() has a mistake.

---

Based on the idea in Tadhg McDonald-Jensen's answer, my currently best solution is the following. Any hints on how to get rid of the __main__. prefix to AttributeError and the reference to attributeErrorCatcher in the traceback would be much appreciated.

>>> def catchAttributeErrors(func):
...     AttributeError_org = AttributeError
...     def attributeErrorCatcher(*args, **kwargs):
...         try:
...             return func(*args, **kwargs)
...         except AttributeError_org as e:
...             import sys
...             class AttributeError(Exception):
...                 pass
...             etype, value, tb = sys.exc_info()
...             raise AttributeError(e).with_traceback(tb.tb_next) from None
...     return attributeErrorCatcher
...
>>> def deeply_nested_factory_fn():
...     a = 2
...     return a.invalid_attr
...
>>> class Test(object):
...     def __getattr__(self, name):
...         if name == 'abc':
...             # computing come other attributes
...             return 'abc'
...         raise AttributeError("'Test' object has no attribute '%s'" % name)
...     @property
...     @catchAttributeErrors
...     def my_prop(self):
...         return deeply_nested_factory_fn()
...
>>> class Test1(object):
...     def __init__(self):
...         test = Test()
...         test.my_prop
...
>>> test1 = Test1()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in __init__
  File "<stdin>", line 11, in attributeErrorCatcher
  File "<stdin>", line 10, in my_prop
  File "<stdin>", line 3, in deeply_nested_factory_fn
__main__.AttributeError: 'int' object has no attribute 'invalid_attr'

Christmastide

you would have to do __qualname__ = "AttributeError" in the class definition to remove the __main__ part but trust me, you do not want the error to just say AttributeError because then you risk being baffled why the heck except AttributeError didn't catch an AttributeError. – Coastal

Open issue: github.com/python/cpython/issues/103936 – Anility

Answer 1

Just in case others find this: the problem with the example on top is that an AttributeError is raised inside __getattr__. Instead, one should call self.__getattribute__(attr) to let that raise.

Example

def deeply_nested_factory_fn():
    a = 2
    return a.invalid_attr

class Test(object):
    def __getattr__(self, name):
        if name == 'abc':
            return 'abc'
        return self.__getattribute__(name)
    @property
    def my_prop(self):
        return deeply_nested_factory_fn()

test = Test()
test.my_prop

This yields

AttributeError                            Traceback (most recent call last)
Cell In [1], line 15
     12         return deeply_nested_factory_fn()
     14 test = Test()
---> 15 test.my_prop

Cell In [1], line 9, in Test.__getattr__(self, name)
      7 if name == 'abc':
      8     return 'abc'
----> 9 return self.__getattribute__(name)

Cell In [1], line 12, in Test.my_prop(self)
     10 @property
     11 def my_prop(self):
---> 12     return deeply_nested_factory_fn()

Cell In [1], line 3, in deeply_nested_factory_fn()
      1 def deeply_nested_factory_fn():
      2     a = 2
----> 3     return a.invalid_attr

AttributeError: 'int' object has no attribute 'invalid_attr'

Athal

Answer 2

If you're willing to exclusively use new-style classes, you could overload __getattribute__ instead of __getattr__:

class Test(object):
    def __getattribute__(self, name):
        if name == 'abc':
            return 'abc'
        else:
            return object.__getattribute__(self, name)
    @property
    def my_prop(self):
        return deeply_nested_factory_fn()

Now your stack trace will properly mention deeply_nested_factory_fn.

Traceback (most recent call last):
  File "C:\python\myprogram.py", line 16, in <module>
    test.my_prop
  File "C:\python\myprogram.py", line 10, in __getattribute__
    return object.__getattribute__(self, name)
  File "C:\python\myprogram.py", line 13, in my_prop
    return deeply_nested_factory_fn()
  File "C:\python\myprogram.py", line 3, in deeply_nested_factory_fn
    return a.invalid_attr
AttributeError: 'int' object has no attribute 'invalid_attr'

Pressley

Thanks, my issue with this solution is the performance hit for other attributes. (Some of which may be accessed in loops.) – Christmastide

@ARF, if performance is a concern then get local references to the frequently accessed attributes. I'd recommend doing that even without the slight slowing down by overloading __getattribute__. – Coastal

You mean by a = test.a and then using a instead of test.a? – Christmastide

@Christmastide yes, I actually advertised Kevin's answer in mine because it is by far superior to mine. – Coastal

Answer 3

You can create a custom Exception that appears to be an AttributeError but will not trigger __getattr__ since it is not actually an AttributeError.

UPDATED: the traceback message is greatly improved by reassigning the .__traceback__ attribute before re-raising the error:

class AttributeError_alt(Exception):
    @classmethod
    def wrapper(err_type, f):
        """wraps a function to reraise an AttributeError as the alternate type"""
        @functools.wraps(f)
        def alt_AttrError_wrapper(*args,**kw):
            try:
                return f(*args,**kw)
            except AttributeError as e:
                new_err = err_type(e)
                new_err.__traceback__ = e.__traceback__.tb_next
                raise new_err from None
        return alt_AttrError_wrapper

Then when you define your property as:

@property
@AttributeError_alt.wrapper
def my_prop(self):
    return deeply_nested_factory_fn()

and the error message you will get will look like this:

Traceback (most recent call last):
  File ".../test.py", line 34, in <module>
    test.my_prop
  File ".../test.py", line 14, in alt_AttrError_wrapper
    raise new_err from None
  File ".../test.py", line 30, in my_prop
    return deeply_nested_factory_fn()
  File ".../test.py", line 20, in deeply_nested_factory_fn
    return a.invalid_attr
AttributeError_alt: 'int' object has no attribute 'invalid_attr'

notice there is a line for raise new_err from None but it is above the lines from within the property call. There would also be a line for return f(*args,**kw) but that is omitted with .tb_next.

---

I am fairly sure the best solution to your problem has already been suggested and you can see the previous revision of my answer for why I think it is the best option. Although honestly if there is an error that is incorrectly being suppressed then raise a bloody RuntimeError chained to the one that would be hidden otherwise:

def assert_no_AttributeError(f):
    @functools.wraps(f)
    def assert_no_AttrError_wrapper(*args,**kw):
        try:
            return f(*args,**kw)
        except AttributeError as e:
            e.__traceback__ = e.__traceback__.tb_next
            raise RuntimeError("AttributeError was incorrectly raised") from e
    return assert_no_AttrError_wrapper

then if you decorate your property with this you will get an error like this:

Traceback (most recent call last):
  File ".../test.py", line 27, in my_prop
    return deeply_nested_factory_fn()
  File ".../test.py", line 17, in deeply_nested_factory_fn
    return a.invalid_attr
AttributeError: 'int' object has no attribute 'invalid_attr'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../test.py", line 32, in <module>
    x.my_prop
  File ".../test.py", line 11, in assert_no_AttrError_wrapper
    raise RuntimeError("AttributeError was incorrectly raised") from e
RuntimeError: AttributeError was incorrectly raised

Although if you expect more then just one thing to raise an AttributeError then you might want to just overload __getattribute__ to check for any peculiar error for all lookups:

def __getattribute__(self,attr):
    try:
        return object.__getattribute__(self,attr)
    except AttributeError as e:
        if str(e) == "{0.__class__.__name__!r} object has no attribute {1!r}".format(self,attr):
            raise #normal case of "attribute not found"
        else: #if the error message was anything else then it *causes* a RuntimeError
            raise RuntimeError("Unexpected AttributeError") from e

This way when something goes wrong that you are not expecting you will know it right away!

Coastal

This does not really make things much better: The stack trace is still way off! Note that the error is is in function deeply_nested_factory_fn not in function inner as suggested by the stack trace produced by your solution. - That said, I would really like your idea which avoids sys.exit(1)... if the stacktrace issue could be fixed. – Christmastide

After some head-scratching, I came up with a solution I am not too unhappy about. If you are still interested, have a look as the modified question. – Christmastide

This is going to stop people from doing except AttributeError to catch this exception. – Sensation

@Sensation That is an excellent point. I will have to think about things a bit more. – Christmastide

@TadhgMcDonald-Jensen Actually, for my application the hasattr behaviour you illustrate is desirable. I am having the issue that hasattr fails "silently" in a if hasattr(...): .... else: .... block - not because the property is undefined but because the implementation has a bug. Of course that other piece of code should really avoid hasattr altogether and just use a try: some_obj.attr except AttributeError: .... block instead... – Christmastide

@TadhgMcDonald-Jensen To avoid the issue alltogether it would be nice if one could "re-raise" the attribute error in __getattr__ depending on whether the AttributeError originated in the property on not. But I think, once we get to __getattr__ all traces of the AttributeError from the property are gone. Or is there a way to recover this error and its stack trace somehow? – Christmastide

I edited my answer to suggest exception chaining just like I did with this question. And I'll tell you what I told him: Faking a seamless error message is not something you want to do. You want an error that tells you what actually happened in execution and be happy that python gives such informative traceback messages. – Coastal

Thanks for the RuntimeError chaining suggestion. It looks like a simple, clean solution for my particular situation. – Christmastide