Read and merge two Yaml files in go language

Question

Assuming we have two yaml files

master.yaml

someProperty: "someVaue"
anotherProperty: "anotherValue"

override.yaml

someProperty: "overriddenVaue"

Is it possible to unmarshall, merge, and then write those changes to a file without having to define a struct for every property in the yaml file?

The master file has over 500 properties in it that are not at all important to the service at this point of execution, so ideally I'd be able to just unmarshal into a map, do a merge and write out in yaml again but I'm relatively new to go so wanted some opinions.

I've got some code to read the yaml into an interface but i'm unsure on the best approach to then merge the two.

var masterYaml interface{}
yamlBytes, _ := ioutil.ReadFile("master.yaml")
yaml.Unmarshal(yamlBytes, &masterYaml)

var overrideYaml interface{}
yamlBytes, _ = ioutil.ReadFile("override.yaml")
yaml.Unmarshal(yamlBytes, &overrideYaml)

I've looked into libraries like mergo but i'm not sure if that's the right approach.

I'm hoping that after the master I would be able to write out to file with properties

someProperty: "overriddenVaue"
anotherProperty: "anotherValue"

Mandi

I'm sure it is possible. What have you tried? What problems did you encounter? – Clientage

It is possible without defining the struct but I cannot understand what you actually want to merge which fields you want to merge and how. – Directoire

Why don't you use struct with only the fields you required and skip the fields you do not require that will save a lot of work using interfaces. – Directoire

I've added some extra information. I'm not sure which fields may be overridden, they are fields that could be passed to us at will, which is why I was hoping we'd be able to be super generic and abstract about this. – Mandi

this is really interesting question, i've posted answer for deep merge #51948379 – Outlast

Answer 1

Assuming that you just want to merge at the top level, you can unmarshal into maps of type map[string]interface{}, as follows:

package main

import (
    "io/ioutil"

    "gopkg.in/yaml.v2"
)

func main() {
    var master map[string]interface{}
    bs, err := ioutil.ReadFile("master.yaml")
    if err != nil {
        panic(err)
    }
    if err := yaml.Unmarshal(bs, &master); err != nil {
        panic(err)
    }

    var override map[string]interface{}
    bs, err = ioutil.ReadFile("override.yaml")
    if err != nil {
        panic(err)
    }
    if err := yaml.Unmarshal(bs, &override); err != nil {
        panic(err)
    }

    for k, v := range override {
        master[k] = v
    }

    bs, err = yaml.Marshal(master)
    if err != nil {
        panic(err)
    }
    if err := ioutil.WriteFile("merged.yaml", bs, 0644); err != nil {
        panic(err)
    }
}

Congou

Oh snap, map[string]interface{} and then iterating as a map is a perfect solution. Cheers. – Mandi

How might this look multiple levels deep? Is an approach like if reflect.ValueOf(masterYaml[k]).Kind() == reflect.Map recommended in Go? – Mandi

I wouldn't recommend reflection, but you can do things like if m, ok := v.(map[interface{}]interface{}); ok { ... and treat the case of maps specially. – Congou

this doesn't handle nested merging – Feudal

Answer 2

For a broader solution (with n input files), you can use this function. I have used @robox answer to do my solution:

func ReadValues(filenames ...string) (string, error) {
    if len(filenames) <= 0 {
        return "", errors.New("You must provide at least one filename for reading Values")
    }
    var resultValues map[string]interface{}
    for _, filename := range filenames {

        var override map[string]interface{}
        bs, err := ioutil.ReadFile(filename)
        if err != nil {
            log.Info(err)
            continue
        }
        if err := yaml.Unmarshal(bs, &override); err != nil {
            log.Info(err)
            continue
        }

        //check if is nil. This will only happen for the first filename
        if resultValues == nil {
            resultValues = override
        } else {
            for k, v := range override {
                resultValues[k] = v
            }
        }

    }
    bs, err := yaml.Marshal(resultValues)
    if err != nil {
        log.Info(err)
        return "", err
    }

    return string(bs), nil
}

So for this example you should call it with this order:

result, _ := ReadValues("master.yaml", "overwrite.yaml")

In the case you have an extra file newFile.yaml, you could also use this function:

result, _ := ReadValues("master.yaml", "overwrite.yaml", "newFile.yaml")

Camisole

Answer 3

DEEP MERGE TWO YAML FILES

package main

import (
    "fmt"
    "io/ioutil"
    "sigs.k8s.io/yaml"
)

func main() {

    // declare two map to hold the yaml content
    base := map[string]interface{}{}
    currentMap := map[string]interface{}{}

    // read one yaml file 
    data, _ := ioutil.ReadFile("conf.yaml")
    if err := yaml.Unmarshal(data, &base); err != nil {

    }

    // read another yaml file
    data1, _ := ioutil.ReadFile("conf1.yaml")
    if err := yaml.Unmarshal(data1, &currentMap); err != nil {

    }

    // merge both yaml data recursively
    base = mergeMaps(base, currentMap)

    // print merged map
    fmt.Println(base)

}

func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
    out := make(map[string]interface{}, len(a))
    for k, v := range a {
        out[k] = v
    }
    for k, v := range b {
        if v, ok := v.(map[string]interface{}); ok {
            if bv, ok := out[k]; ok {
                if bv, ok := bv.(map[string]interface{}); ok {
                    out[k] = mergeMaps(bv, v)
                    continue
                }
            }
        }
        out[k] = v
    }
    return out
}

Outlast