Can I insert a line into ruamel.yaml's CommentedMap?

Question

I understand that this is related to this SO question, but what I'm mostly concerned about is whether this might mess with things such as the preserved comments.

import ruamel.yaml as yaml

yaml_str = """\
first_name: Art
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...
"""

data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)

# I'd like to extend CommentedMap so that I can do something like:
data.insert(1, 'last_name', 'Vandelay')

print(yaml.dump(data, Dumper=yaml.RoundTripDumper))

Should output:

first_name: Art
last_name: Vandelay
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...

Should not output:

first_name: Art
last_name: Vandelay    # This is an occupation comment
occupation: Architect
about: Art Vandelay is a fictional character that George invents...

Scarf

Answer 1

On Python 2.7, and on Python 3.X with at least ruamel.yaml 0.11.11, this works fine:

import ruamel.yaml

yaml_str = """\
first_name: Art
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...
"""

data = ruamel.yaml.round_trip_load(yaml_str)
data.insert(1, 'last name', 'Vandelay')

print(ruamel.yaml.round_trip_dump(data))

gives:

first_name: Art
last name: Vandelay
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...

as the end-of-line comments are associated with the key of the line in the CommentedMap. (Python 2.7.11 on Linux Mint with ruamel.yaml 0.11.10.)

This will not work on older versions of ruamel.yaml with Python3 as the .insert() you are using is a feature of the full-fledged ruamel.ordereddict and the OrderedDict in the standard library doesn't have that method. Therefore you need to graft an .insert() function onto the CommentedMap:

import ruamel.yaml
from ruamel.yaml.comments import CommentedMap
from ruamel.yaml.compat import ordereddict

yaml_str = """\
first_name: Art
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...
"""

def com_insert(self, pos, key, value, comment=None):
    od = ordereddict()
    od.update(self)
    for k in od:
        del self[k]
    for index, old_key in enumerate(od):
        if pos == index:
            self[key] = value
        self[old_key] = od[old_key]
    if comment is not None:
        self.yaml_add_eol_comment(comment, key=key)

CommentedMap.insert = com_insert

data = ruamel.yaml.round_trip_load(yaml_str)
data.insert(1, 'last name', 'Vandelay', comment="new key")

print(ruamel.yaml.round_trip_dump(data))

gives on Python3:

first_name: Art
last name: Vandelay    # new key
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...

Please note that there is an optional parameter for insert() that allows you to specify a comment for the newly inserted key-value pair. The above works because deleting a key from a CommentedMap doesn't remove the comment associated with the key. So I temporarily park the old key-value pairs in od delete all key-values, and then copy them back inserting the new stuff at the right moment

The above insert, with comment, has been added in ruamel.yaml 0.11.11 for both Python 2 and 3

---

The .round_trip_load() is equivalent to .load(...., Loader=ruamel.yaml.RoundTripLoader, ...) and .round_trip_dump() to `.dump(....., Dumper=ruamel.yaml.RoundTripDumper, allow_unicode=True, ...)

Substitute

wait what?? I had no idea ruamel.ordereddict had .insert() :D – Scarf

That is excusable, the author of that package is notoriously lazy when it comes to writing accessible documentation. – Substitute

I'm just happy that it's available at all. Thank you! But can I somehow use ruamel.ordereddict even though I'm using Python 3? – Scarf

Looking at compat.py, line 12 it seems all I have to do is pip install ruamel.ordereddict. Is this a correct analysis? – Scarf

Unfortunately not. ruamel.ordereddict only works on Python2. I got most of it working on 3.2 (after that there were rather big changes in the C dict source that I tried to follow as close as possible). But then found that for 3.5 a C based OrderedDict was being done from scratch and decided not to spent more time on that and focus on other things. – Substitute

So what you would need to do is create a new CommentedMap, copy the key, value pairs over with any comments and insert the new key at the right moment. I can look at extending the answer with that tomorrow. – Substitute

Yeah, if you have the time, that would be super helpful! Thanks! – Scarf

This is now in ruamel.yaml 0.11.11 and documented. Thanks for the inspiration. – Substitute

What I created with this is a hook for Python Schematics The goal is to reformat the yaml as little as possible when saving back but to also have a basic structure for the data when new data is added. – Scarf