How to read class attributes in the same order as declared?

Question

I am writing a metaclass that reads class attributes and store them in a list, but I want the list (cls.columns) to respect the declaration order (that is: mycol2, mycol3, zut, cool, menfin, a in my example):

import inspect
import pprint

class Column(object):
    pass

class ListingMeta(type):
    def __new__(meta, classname, bases, classDict):
        cls = type.__new__(meta, classname, bases, classDict)
        cls.columns = inspect.getmembers(cls, lambda o: isinstance(o, Column)) 
        cls.nb_columns = len(cls.columns)
        return cls

class Listing(object):
    __metaclass__ = ListingMeta
    mycol2 = Column()
    mycol3 = Column()
    zut = Column()
    cool = Column()
    menfin = Column()
    a = Column()

pprint.pprint(Listing.columns)

Result:

[('a', <__main__.Column object at 0xb7449d2c>),
 ('cool', <__main__.Column object at 0xb7449aac>),
 ('menfin', <__main__.Column object at 0xb7449a8c>),
 ('mycol2', <__main__.Column object at 0xb73a3b4c>),
 ('mycol3', <__main__.Column object at 0xb744914c>),
 ('zut', <__main__.Column object at 0xb74490cc>)]

This does not respect the declaration order of Column() attributes for Listing class. If I use classDict directly, it does not help either.

How can I proceed?

Humidistat

I don't think you can get them in order without some sort of source-level analysis. In any case, the order is supposed to be mostly irrelevant. The dict hashes by key, which is why you're not seeing it in order – Callaghan

At all, a very constructive question. thanks – Prescriptive

you can take a look on tosca widget 2, to find how to do that – Prescriptive

Answer 1

In the current version of Python, the class ordering is preserved. See PEP520 for details.

In older versions of the language (3.5 and below, but not 2.x), you can provide a metaclass which uses an OrderedDict for the class namespace.

import collections 

class OrderedClassMembers(type):
    @classmethod
    def __prepare__(self, name, bases):
        return collections.OrderedDict()

    def __new__(self, name, bases, classdict):
        classdict['__ordered__'] = [key for key in classdict.keys()
                if key not in ('__module__', '__qualname__')]
        return type.__new__(self, name, bases, classdict)

class Something(metaclass=OrderedClassMembers):
    A_CONSTANT = 1

    def first(self):
        ...

    def second(self):
        ...

print(Something.__ordered__)
# ['A_CONSTANT', 'first', 'second']

This approach doesn't help you with existing classes, however, where you'll need to use introspection.

Soundboard

Any ideas or pointers as how to use and learn about introspection? (Or better, how to apply it in this case.) – Reisinger

It should also be noted that even in P3.6 the order is not always retained when using external classes to help define the Something class items. – Reisinger

Answer 2

Here is the workaround I juste developped :

import inspect

class Column(object):
    creation_counter = 0
    def __init__(self):
        self.creation_order = Column.creation_counter
        Column.creation_counter+=1

class ListingMeta(type):
    def __new__(meta, classname, bases, classDict):
        cls = type.__new__(meta, classname, bases, classDict)
        cls.columns = sorted(inspect.getmembers(cls,lambda o:isinstance(o,Column)),key=lambda i:i[1].creation_order) 
        cls.nb_columns = len(cls.columns)
        return cls

class Listing(object):
    __metaclass__ = ListingMeta
    mycol2 = Column()
    mycol3 = Column()
    zut = Column()
    cool = Column()
    menfin = Column()
    a = Column()


for colname,col in Listing.columns:
    print colname,'=>',col.creation_order

Humidistat

First I thought "You must reset the creation_counter after each class", and then I realized that you don't at all, assuming you only care about the internal order. It actually works. :) – Rodarte

How it working in parallel threads ? i think this code is not threadsafe. – Prescriptive

For Python3.6+ refer to @Conchylicultor answer, which is likely based on newer dict order insertion guarantees by the language. You can use vars(cls) rather than the uglier cls.__dict__. – Fudge

Answer 3

For python 3.6, this has become the default behavior. See PEP520: https://www.python.org/dev/peps/pep-0520/

class OrderPreserved:
    a = 1
    b = 2
    def meth(self): pass

print(list(OrderPreserved.__dict__.keys()))
# ['__module__', 'a', 'b', 'meth', '__dict__', '__weakref__', '__doc__']

Gustafsson

Answer 4

1) Since Python 3.6 attributes in a class definition have the same order in which the names appear in the source. This order is now preserved in the new class’s __dict__ attribute (https://docs.python.org/3.6/whatsnew/3.6.html#whatsnew36-pep520):

class Column:
    pass

class MyClass:
    mycol2 = Column()
    mycol3 = Column()
    zut = Column()
    cool = Column()
    menfin = Column()
    a = Column()

print(MyClass.__dict__.keys())

You will see output like this (MyClass.__dict__ may be used like OrderedDict):

dict_keys(['__module__', 'mycol2', 'mycol3', 'zut', 'cool', 'menfin', 'a', '__dict__', '__weakref__', '__doc__'])

Note extra __xxx__ fields added by python, you may need to ignore them.

2) For previous Python 3.x versions you can use solution based on by @Duncan answer, but simpler. We use that fact, that __prepare__ method returns a OrderDict instead of simple dict - so all attributes gathered before __new__ call will be ordered.

from collections import OrderedDict

class OrderedClass(type):
    @classmethod
    def __prepare__(mcs, name, bases): 
         return OrderedDict()

    def __new__(cls, name, bases, classdict):
        result = type.__new__(cls, name, bases, dict(classdict))
        result.__fields__ = list(classdict.keys())
        return result

class Column:
    pass

class MyClass(metaclass=OrderedClass):
    mycol2 = Column()
    mycol3 = Column()
    zut = Column()
    cool = Column()
    menfin = Column()
    a = Column()

Now you can use attribute __fields__ for accessing attributes in required order:

m = MyClass()
print(m.__fields__)
['__module__', '__qualname__', 'mycol2', 'mycol3', 'zut', 'cool', 'menfin', 'a']

Note that there will be attrs '__module__', '__qualname__' born from type class. To get rid of them you may filter names in following manner (change OrderedClass.__new__):

def __new__(cls, name, bases, classdict):
    result = type.__new__(cls, name, bases, dict(classdict))
    exclude = set(dir(type))
    result.__fields__ = list(f for f in classdict.keys() if f not in exclude)
    return result    

it will give only attrs from MyClass:

['mycol2', 'mycol3', 'zut', 'cool', 'menfin', 'a']

3) this anwser is only workable in python3.x, because there is no __prepare__ definition in python2.7

Savor

Answer 5

If you are using Python 2.x then you'll need a hack such as the one Lennart proposes. If you are using Python 3.x then read PEP 3115 as that contains an example which does what you want. Just modify the example to only look at your Column() instances:

 # The custom dictionary
 class member_table(dict):
    def __init__(self):
       self.member_names = []

    def __setitem__(self, key, value):
       # if the key is not already defined, add to the
       # list of keys.
       if key not in self:
          self.member_names.append(key)

       # Call superclass
       dict.__setitem__(self, key, value)

 # The metaclass
 class OrderedClass(type):

     # The prepare function
     @classmethod
     def __prepare__(metacls, name, bases): # No keywords in this case
        return member_table()

     # The metaclass invocation
     def __new__(cls, name, bases, classdict):
        # Note that we replace the classdict with a regular
        # dict before passing it to the superclass, so that we
        # don't continue to record member names after the class
        # has been created.
        result = type.__new__(cls, name, bases, dict(classdict))
        result.member_names = classdict.member_names
        return result

 class MyClass(metaclass=OrderedClass):
    # method1 goes in array element 0
    def method1(self):
       pass

    # method2 goes in array element 1
    def method2(self):
       pass

Crackerbarrel

Answer 6

An answer that excludes methods:

from collections import OrderedDict
from types import FunctionType


class StaticOrderHelper(type):
    # Requires python3.
    def __prepare__(name, bases, **kwargs):
        return OrderedDict()

    def __new__(mcls, name, bases, namespace, **kwargs):
        namespace['_field_order'] = [
                k
                for k, v in namespace.items()
                if not k.startswith('__') and not k.endswith('__')
                    and not isinstance(v, (FunctionType, classmethod, staticmethod))
        ]
        return type.__new__(mcls, name, bases, namespace, **kwargs)


class Person(metaclass=StaticOrderHelper):
    first_name = 'First Name'
    last_name = 'Last Name'
    phone_number = '000-000'

    @classmethod
    def classmethods_not_included(self):
        pass

    @staticmethod
    def staticmethods_not_included(self):
        pass

    def methods_not_included(self):
        pass


print(Person._field_order)

Disable

This also works for using imported Scrapy items when they are assigned in a class with some_item = Field(). – Reisinger

Answer 7

I guess you should be able to make a class where you replace its __dict__ with an ordered-dict

Penstemon

Nice idea. have you tested it ? – Prescriptive

I tried but it turned out to be not in defined order – Peeples