How to update yaml file using python

Question

I have a some.yaml file with the below contents.

    init_config: {}
    instances:
        - host: <IP>
          username: <username>
          password: <password>

The yaml file should be parsed and updated as below.

    init_config: {}
    instances:
        - host: 1.2.3.4
          username: Username
          password: Password

How do I parse the values and update them appropriately?

Bearden

If you use PyYaml, you can use Loader to load data, and Dumper to write data to file. The data loaded is an ordinary dictionary in Python so you can access element by key and thus change it as you wish. – Hohenstaufen

It is not clear if you want a template engine (and are misdirecting useful answers by being too narrow) or simply want to parse a YAML and substitute some string values on the parsed data. Be more specific and explain what you plain to achieve and why. – Furnishing

It is a file. I need to parse a yaml file and replace the contents. – Bearden

Why you need to parse the YAML if you seem only interested in replacing some content? Are you interested in the actual data struct, or only want the resulting file? – Furnishing

I am new to yaml. My intention is to replace the text. I used word "parse" meaning reading the file Sorry if that has different purpose – Bearden

Answer 1

The ruamel.yaml package was specifically enhanced (by me starting from PyYAML) to do this kind of round-trip, programmatic, updating.

If you start with (please note I removed the extra initial spaces):

init_config: {}
instances:
    - host: <IP>              # update with IP
      username: <username>    # update with user name
      password: <password>    # update with password

and run:

import ruamel.yaml

file_name = 'input.yaml'
config, ind, bsi = ruamel.yaml.util.load_yaml_guess_indent(open(file_name))

instances = config['instances']
instances[0]['host'] = '1.2.3.4'
instances[0]['username'] = 'Username'
instances[0]['password'] = 'Password'

yaml = ruamel.yaml.YAML()
yaml.indent(mapping=ind, sequence=ind, offset=bsi) 
with open('output.yaml', 'w') as fp:
    yaml.dump(config, fp)

The output will be:

init_config: {}
instances:
    - host: 1.2.3.4           # update with IP
      username: Username      # update with user name
      password: Password      # update with password

The ordering of mapping keys (host, username and password), the style and the comments are preserved without any further specific action.

Instead of having the indent and block sequence indent guessed, you can do a manual traditional load, and set the indent values yourself:

yaml = ruamel.yaml.YAML()
yaml.indent(mapping=6, sequence=4)
with open(file_name) as fp:
    config = yaml.load(fp)

---

If you look at the history of this answer, you can see how to do this with a more limited, PyYAML like, API.

Apery

Hi, I've noticed that loading and dumping has changed in the ruamel.yaml package, so you might want to update this answer. – Quant

It seems to have changed again – Megmega

@Megmega Can you elaborate your statement into something useful? – Apery

Of course, sorry. I tried following your example, but load_yaml_guess_indent doesn't seem to exist in ruamel.yaml anymore. If that method was from another library than ruamel, perhaps you could update the example to indicate that? Either way, the code won't run now because load_yaml_guess_indent() is called as a top-level function which doesn't exist. That said, thanks for pointing me in the right direction so I could get my own problem solved! – Megmega

@Megmega import ruamel.yaml was never enough to get the "guessing" function imported, I had forgotten to copy another line from my test program. I now solved this differently, and it should now run. – Apery

@alper I am not sure what "single key" is, if you intended to write "a single key", I am not sure if you mean changing a key, but retaining its value (which is possible, but since comments are assocated with keys, not have the effect you want without extra work). What have you tried that did not work? I suggest you post a complete question on Stack Overflow, instead of what to me isan incomplete sentence as comment. – Apery

I meant to change only a single key's value and keep the other variables as it is – Lala

@Lala There are no variables in a YAML file. Stil not sure what you try to do and what is not working. The above program loads YAML into a Python datai structure then updates the values for three keys in a single dict nested within that data structure. If you only want to update one of those, I would think it obvious you can when you comment out the other two lines. But that is so trivial, I think you want to do something different. – Apery

Sorry for not being clear. As I understand yaml.dump() overwrites to file from scratch like, cat "hello" > file.txt does. Instead I was just wondering would it be possible to , only change the corresponding updated value section to be more efficient. – Lala

Answer 2

This is how i can read from the above file i mentioned, parse and update as needed.

import yaml

fname = "some.yaml"

stream = open(fname, 'r')
data = yaml.load(stream)

data['instances'][0]['host'] = '1.2.3.4'
data['instances'][0]['username'] = 'Username'
data['instances'][0]['password'] = 'Password'

with open(fname, 'w') as yaml_file:
    yaml_file.write( yaml.dump(data, default_flow_style=False))

Bearden

You should probably mention that this strips the leading spaces, and is not guaranteed to preserve the ordering of the mapping keys (when I run this I get host, password, username instead of host, username, 'password'). – Apery

Use yaml.load(stream, Loader=yaml.FullLoader) if you trust the source of your YAML file. See <github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation> for details – Chivalric

Anthon, I also agree with Besi. I had the same problem. It´s the Loader. #12013274 – Lauer

Answer 3

I don't know if you need YAML. Aside from using the YAML tag, it seems that you have no interest in the YAML document. So why not using Jinja2 or some template language?

from jinja2 import Template

tmpl = Template(u'''\
    init_config: {}
    instances:
         - host: {{ IP }}
           username: {{ username }}
           password: {{ password }}
''')

print tmpl.render(
     IP=u"1.2.3.4",
     username=u"Username",
     password=u"Password"
)

I don't know if it is a good idea, but if you only need to obtain a file with some fields changed, you don't need to actually parse the YAML document and can benefit from a Template language directly.

---

Bonus: Use case

I have worked with very complex YAML documents, for which there are tags unknown

...
  propertiesIDs: { 1, 2, 3, 4 }
  globalID: !myapplication.InterfaceID &primitiveID

replication: !myapplication.replication
  beginDate: 2012-09-10T20:00:03
  endDate: 2020-09-10T20:00:04
  replicant_uuid:
    ? 17169504-B6AB-11E4-8437-36E258BB2172
    ? 206B5842-B6AB-11E4-AAC3-36E258BB2172
...

Performing a valid parse of this document is difficult and time-consuming. I only need to populate some values, and the YAML is sent to a third-party application. So instead of parsing the YAML or trying to generate a valid document directly using pyyaml, is simpler (more time-efficient, less bug-prone) to generate it directly through templates. Moreover, template languages can easily be used with loops to populate dynamically sized fields.

Furnishing

Answer 4

Here's how i generate docker-crane templates for dev, production, stage, etc...

1. mkdir crane_templates
2. touch crane_templates/init.py
3. Add template content with nano crane_templates/some.yaml
4. Nano crane_gen.py

--- crane_gen.py ---

#!/usr/bin/env python
from jinja2 import Environment, PackageLoader

env = Environment(loader=PackageLoader('crane_templates', './'))
tmpl = env.get_template('crane.yaml.tmpl')

result = tmpl.render(
     IP=u"1.2.3.4",
     username=u"Username",
     password=u"Password"
)

5. python crane_gen.py > result.yaml

Answer inspired by @MariusSiuram

Airliah

Answer 5

Here are sample using PyYaml. As I understand you have something like template in yaml format, and you have to substitute places in angle brackets with actual values.

import yaml

s = """
    init_config: {}
    instances:
        - host: <IP>
          username: <username>
          password: <password>
"""

dict_obj = yaml.load(s) # loads string in internal data structure - dict
dict_obj['instances'][0]['host'] = 'localhost' # change values
print yaml.dump(dict_obj) # dumps dict to yaml format back

Sweetbread