When ruamel.yaml loads @dataclass from string, __post_init__ is not called

Question

Assume I created a @dataclass class Foo, and added a __post_init__ to perform type checking and processing.

When I attempt to yaml.load a !Foo object, __post_init__ is not called.

from dataclasses import dataclass, fields

from ruamel.yaml import yaml_object, YAML


yaml = YAML()


@yaml_object(yaml)
@dataclass
class Foo:
    foo: int
    bar: int

    def __post_init__(self):
        raise Exception
        for field in fields(self):
            value = getattr(self, field.name)
            typ = field.type
            if not isinstance(value, typ):
                raise Exception

s = '''\
!Foo
foo: "foo"
bar: "bar"
'''
yaml.load(s)

How do I perform parameter checking when loading dataclasses via ruamel.yaml?

This behavior occurs in Python 3.7 as well as 3.6 with pip install dataclasses.

Microwatt

Since I learned Rust, I've started writing static factory methods instead of constructors. Maybe if I had this problem again, I'd approach it differently. Not sure. – Microwatt

Answer 1

This is now supported in 0.17.34, where you can do

from dataclasses import dataclass
from ruamel.yaml import YAML

yaml = ruamel.yaml.YAML()

@yaml.register_class
@dataclass
class Foo:
   ....

---

The reason why __post_init__ is not called, is because ruamel.yaml (and the PyYAML code in its Constructors), was created long before dataclasses was created.

Of course code for making a call to __post_init_() could be added to ruamel.yaml's Python object constructors, preferably after a test if something was created using @dataclass, as otherwise a non Data-Class class, that happens to have such a method named __post_init_, will all of a sudden have that method called during loading.

If you have no such classes, you can add your own, smarter, constructor to the YAML() instance before first loading/dumping (at which moment the constructor is instantiated) using yaml.Constructor = MyConstructor. But adding a constructor is not as trivial as subclassing the RoundTripConstructor, because all supported node types need to be registered on such a new constructor type.

Most of the time I find it easier to just patch the appropriate method on the RoundTripConstructor:

from dataclasses import dataclass, fields
from ruamel.yaml import yaml_object, YAML, RoundTripConstructor


def my_construct_yaml_object(self, node, cls):
    for data in self.org_construct_yaml_object(node, cls):
      yield data
    # not doing a try-except, in case `__post_init__` does catch the AttributeError
    post_init = getattr(data, '__post_init__', None)
    if post_init:
        post_init()

RoundTripConstructor.org_construct_yaml_object = RoundTripConstructor.construct_yaml_object
RoundTripConstructor.construct_yaml_object = my_construct_yaml_object

yaml = YAML()
yaml.preserve_quotes = True

@yaml_object(yaml)
@dataclass
class Foo:
    foo: int
    bar: int

    def __post_init__(self):
        for field in fields(self):
            value = getattr(self, field.name)
            typ = field.type
            if not isinstance(value, typ):
                raise Exception

s = '''\
!Foo
foo: "foo"
bar: "bar"
'''
d = yaml.load(s)

throws an exception:

Traceback (most recent call last):
  File "try.py", line 36, in <module>
    d = yaml.load(s)
  File "/home/venv/tmp-46489abf428c4cd4/lib/python3.7/site-packages/ruamel/yaml/main.py", line 266, in load
    return constructor.get_single_data()
  File "/home/venv/tmp-46489abf428c4cd4/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 105, in get_single_data
    return self.construct_document(node)
  File "/home/venv/tmp-46489abf428c4cd4/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 115, in construct_document
    for dummy in generator:
  File "try.py", line 10, in my_construct_yaml_object
    post_init()
  File "try.py", line 29, in __post_init__
    raise Exception
Exception

Please note that the double quotes in your YAML are superfluous, so if you want to preserve these on round-trip you need to do yaml.preserve_quotes = True

Adelaidaadelaide

Support for __post_init__ has been added to ruamel.yaml as of 0.17.35 sourceforge.net/p/ruamel-yaml/tickets/355 – Pickwickian

Answer 2

I'm not entirely sure if this is the correct workaround...

I can move logic from __post_init__ to __setstate__(state: dict), which gets called by YAML().load().

def __setstate__(self, state):
    self.__dict__.update(state)
    # I could call self.__post_init__(), or alternatively move logic here:
    for field in fields(self):
        value = getattr(self, field.name)
        typ = field.type
        if not isinstance(value, typ):
            raise Exception

YAML().load(s) calls Foo.__setstate__(state) if that method exists, but apparently not __init__ (which calls __post_init__). Is this an intentional design decision?

Microwatt

Yes that is a an intentional design decision, it is similar to what pickle does. And if you pickle.dump your Foo instance (without __post_init__) and then try to pickle.load it with __post_init__, you'll notice it doesn't get called either. Also see my answer here – Adelaidaadelaide