Iterate over object attributes in python [duplicate]

Question

I have a python object with several attributes and methods. I want to iterate over object attributes.

class my_python_obj(object):
    attr1='a'
    attr2='b'
    attr3='c'

    def method1(self, etc, etc):
        #Statements

I want to generate a dictionary containing all of the objects attributes and their current values, but I want to do it in a dynamic way (so if later I add another attribute I don't have to remember to update my function as well).

In php variables can be used as keys, but objects in python are unsuscriptable and if I use the dot notation for this it creates a new attribute with the name of my var, which is not my intent.

Just to make things clearer:

def to_dict(self):
    '''this is what I already have'''
    d={}
    d["attr1"]= self.attr1
    d["attr2"]= self.attr2
    d["attr3"]= self.attr3
    return d

·

def to_dict(self):
    '''this is what I want to do'''
    d={}
    for v in my_python_obj.attributes:
        d[v] = self.v
    return d

Update: With attributes I mean only the variables of this object, not the methods.

Possie

#1252192 Might be of some help. – Unhappy

@Unhappy In particular, https://mcmap.net/q/48347/-how-to-enumerate-an-object-39-s-properties-in-python-duplicate is the way to go; you'd use the optional predicate argument to pass a callable that would filter out (bound?) functions (that gets rid of the methods). – Rhizotomy

Per sean, does this answer your question? How to enumerate an object's properties in Python? – Signalman

Answer 1

Assuming you have a class such as

>>> class Cls(object):
...     foo = 1
...     bar = 'hello'
...     def func(self):
...         return 'call me'
...
>>> obj = Cls()

calling dir on the object gives you back all the attributes of that object, including python special attributes. Although some object attributes are callable, such as methods.

>>> dir(obj)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'bar', 'foo', 'func']

You can always filter out the special methods by using a list comprehension.

>>> [a for a in dir(obj) if not a.startswith('__')]
['bar', 'foo', 'func']

or if you prefer map/filters.

>>> filter(lambda a: not a.startswith('__'), dir(obj))
['bar', 'foo', 'func']

If you want to filter out the methods, you can use the builtin callable as a check.

>>> [a for a in dir(obj) if not a.startswith('__') and not callable(getattr(obj, a))]
['bar', 'foo']

You could also inspect the difference between your class and its instance object using.

>>> set(dir(Cls)) - set(dir(object))
set(['__module__', 'bar', 'func', '__dict__', 'foo', '__weakref__'])

Afoul

This is a bad idea, I wouldn't even suggest it, but if you're going to, at least provide the caveat. – Nautilus

This seems straightforward. What do you mean with caveat? What could go wrong? – Possie

dir() will also list methods. I only want to list variables – Possie

@Pablo methods are also objects, but they're callable so you could exclude any callable objects. I will update the example. – Afoul

@Afoul @Pablo What's wrong is mainly that you shouldn't need to do this. Which attributes you're interested in should be inclusive not exclusive. As you've already seen, you're including methods, and any attempt to exclude them will be flawed, because they're going to involve nasty things like callable or types.MethodType. What happens if I add an attribute called "_foo" that stores some intermediate result, or internal data structure? What happens if I just harmlessly add an attribute to the class, a name, say, and now all of a sudden it gets included. – Nautilus

@Nautilus the code above will pick both _foo and name because they are now attributes of the object. If you don't want to include them you can exclude them in the list comprehension condition. The code above is a common python idiom and popular way of introspecting python objects. – Afoul

not callable is not working.. perhaps @Nautilus is right and I shouldn't be doing this. – Possie

@Afoul That was the point, that it'd include them, and you'd specifically need to go out of your way to exclude them :). Anyways, I can chalk this up to a difference in philosophies, but IMHO good Python code never needs to do this. You use containers, and explicitly declare what you want to include. @Pablo I didn't say it wouldn't appear to work :), as given, what @Afoul posted (including callable certainly should work). – Nautilus

Actually obj.__dict__ is (probably) better for this purpose. – Eury

Yes, __dict__ is an implementation detail. Some objects don't even have one, like objects with __slots__ but still can respond to dir. Neither is ideal though, since dir can also lie – Nautilus

I don't know why SO isn't letting me edit that last comment, but now that you've changed your comment to be suggesting using obj.__dict__, my comment was discouraging it :). – Nautilus

@Pablo callable was not working because I had it wrong. Apologies. I was trying callable(k) where k is just the string. I should have said callable(the_method) which is getattr(obj,k) I have fixed that in my answer now. Apologies again if I have confused you. – Afoul

@Nautilus dir() only "lies" when passed metaclasses (an extreme edge case) or classes with attributes decorated by @DynamicClassAttribute (another extreme edge case). In both cases, calling the dirs() wrapper inspect.getmembers() solves this. For standard objects, however, this solution's list comprehension approach filtering out non-callables absolutely suffices. That some Pythonistas would label it a "bad idea" baffles me, but... to each their own, I suppose? – Yeta

You can also use vars(obj), which return __dict__, instead of dir(obj): https://mcmap.net/q/48348/-how-to-iterate-over-arguments – Chesney

The filter object returns in the filter example. Make sure to put a list method in front of the filter. – Overpraise

@Nautilus If you are using __slots__ with no dict why not just iterate through the slots themselves? – Kassey

Answer 2

in general put a __iter__ method in your class and iterate through the object attributes or put this mixin class in your class.

class IterMixin(object):
    def __iter__(self):
        for attr, value in self.__dict__.iteritems():
            yield attr, value

Your class:

>>> class YourClass(IterMixin): pass
...
>>> yc = YourClass()
>>> yc.one = range(15)
>>> yc.two = 'test'
>>> dict(yc)
{'one': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], 'two': 'test'}

Modulus

this only works when one and two is defined after yc = YourClass(). The question asks about looping through the existing attris in YourClass(). Also, inheriting the IterMixin class may not always available as a solution. – Willi

vars(yc) gives the same result without having to inherit from another class. – Felicio

My above comment isn't quite right; vars(yc) is almost the same, but the dictionary it gives you back is the instance's own __dict__, so modifying it will be reflected on the instance. This can lead to problems if you're not careful, so the above method might sometimes be better (though copying the dictionary is probably easier). Also, note that while the dictionary returned above isn't the instance's __dict__, the values are the same, so modifying any mutable values will still be reflected in the instance's attributes. – Felicio

Answer 3

As mentioned in some of the answers/comments already, Python objects already store a dictionary of their attributes (methods aren't included). This can be accessed as __dict__, but the better way is to use vars (the output is the same, though). Note that modifying this dictionary will modify the attributes on the instance! This can be useful, but also means you should be careful with how you use this dictionary. Here's a quick example:

class A():
    def __init__(self, x=3, y=2, z=5):
        self.x = x
        self._y = y
        self.__z__ = z

    def f(self):
        pass

a = A()
print(vars(a))
# {'x': 3, '_y': 2, '__z__': 5}
# all of the attributes of `a` but no methods!

# note how the dictionary is always up-to-date
a.x = 10
print(vars(a))
# {'x': 10, '_y': 2, '__z__': 5}

# modifying the dictionary modifies the instance attribute
vars(a)["_y"] = 20
print(vars(a))
# {'x': 10, '_y': 20, '__z__': 5}

Using dir(a) is an odd, if not outright bad, approach to this problem. It's good if you really needed to iterate over all attributes and methods of the class (including the special methods like __init__). However, this doesn't seem to be what you want, and even the accepted answer goes about this poorly by applying some brittle filtering to try to remove methods and leave just the attributes; you can see how this would fail for the class A defined above.

(using __dict__ has been done in a couple of answers, but they all define unnecessary methods instead of using it directly. Only a comment suggests to use vars).

Felicio

Something I haven't figured out with the vars() approach is how to handle the situation where the class A has a member that is another object whose type is a user-defined class B. vars(a) seems to call __repr__() on its member of type B. __repr__(), as I understand, is supposed to return a string. But when I call vars(a), it seems like it would make sense for this call to return a nested dict, instead of a dict with a string representation of B. – Dorfman

If you have a.b as some custom class B then vars(a)["b"] is a.b, as one would expect; nothing else would really make sense (think of a.b as syntactic sugar for a.__dict__["b"]). If you have d = vars(a) and call repr(d) then it will call repr(d["b"]) as part of returning it's own repr-string as only class B really knows how it ought to be represented as a string. – Felicio

I tried to use vars() on the svmem object returned by psutil and it failed, saying that __dict__ is not a member of the object. dir() works though, and I can see in its output that __dict__ is not there. psutil docs suggest using ._fields. Just to note a case where vars() failed, although I liked the idea of using it. – Antoneantonella

@KadirA.Peker that's true, __dict__ does need to be defined for this to work. The vast majority of non-built-in Python classes should have this, but for more performance-critical classes, they may define __slots__ instead. Calling dir will always work, but you'll need to filter out the methods and things like __doc__. You could probably also do something like {name: getattr(a, name) for name in a.__slots__} for some object a that has __slots__ defined instead of __dict__. (objects like tuples have neither __dict__ nor __slots__ though) – Felicio

Answer 4

Objects in python store their atributes (including functions) in a dict called __dict__. You can (but generally shouldn't) use this to access the attributes directly. If you just want a list, you can also call dir(obj), which returns an iterable with all the attribute names, which you could then pass to getattr.

However, needing to do anything with the names of the variables is usually bad design. Why not keep them in a collection?

class Foo(object):
    def __init__(self, **values):
        self.special_values = values

You can then iterate over the keys with for key in obj.special_values:

Redcoat

Can you explain a little more how would I use the collection to achieve what I want? – Possie

I'm not adding attributes to the object dynamically. The variables I have will stay the same for a long time, however I think it would be nice to be able to do something like what I intend to. – Possie

Answer 5

class SomeClass:
    x = 1
    y = 2
    z = 3
    
    def __init__(self):
        self.current_idx = 0
        self.items = ["x", "y", "z"]
            
    def next(self):
        if self.current_idx < len(self.items):
            self.current_idx += 1
            k = self.items[self.current_idx-1]
            return (k, getattr(self, k))
        else:
            raise StopIteration
            
    def __iter__(self):
        return self

then just call it as an iterable

s = SomeClass()
for k, v in s:
    print k, "=", v

Ohl

This seems too complicated. If I have no choice, I would rather stick with what i'm doing right now. – Possie

Answer 6

For python 3.6

class SomeClass:

    def attr_list(self, should_print=False):

        items = self.__dict__.items()
        if should_print:
            [print(f"attribute: {k}    value: {v}") for k, v in items]

        return items

Abscind

Can you add some explanation to your posts, so that also non-professional pythonists can benefit from your answers? – Mercurochrome

Answer 7

The correct answer to this is that you shouldn't. If you want this type of thing either just use a dict, or you'll need to explicitly add attributes to some container. You can automate that by learning about decorators.

In particular, by the way, method1 in your example is just as good of an attribute.

Nautilus

Yeah, I know I shouldn't.. but it helps me understand how things work. I come from a php background and used to do this kind of things daily – Possie

You've convinced me. Keeping the to_dict() method updated is far more straightforward than any answer so far. – Possie

Are there any people left that will refrain from dogmatic advice. Dynamic programming will burn you if you're not careful, but the features are in the language to be used. – Barleycorn

I'm sorry, but we are here to answer questions not judge them. – Topgallant

That's not why I'm here, though maybe it's why others are; I'm here to try and help people write good programs, not mindlessly answer questions without concern for whether they're helping people move in that direction. Feel free to treat the site as you would like though. – Nautilus