Go viper .yaml values environment variables override

Question

I'm trying to have application.yaml file in go application which contains ${RMQ_HOST} values which I want to override with environment variables.

In application.yaml I've got:

rmq:
  test:
    host: ${RMQ_HOST}
    port: ${RMQ_PORT}

And in my loader I have:

log.Println("Loading config...")
viper.SetConfigName("application")
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
viper.AutomaticEnv()
err := viper.ReadInConfig()

The problem I have is ${RMQ_HOST} won't get replaced by the values I've set in my environment variables, and tries to connect to the RabbitMQ with this string

amqp://test:test@${RMQ_HOST}:${RMQ_PORT}/test

instead of

amqp://test:test@test:test/test

Photoemission

why use a yaml file when you want to use environmetn variables? – Bergess

Because yaml file values get mapped to environment variables avoiding hardcoded keys making it more extendable over multiple deployment environments. In other words, trying to achieve spring boot functionality. – Photoemission

then i might be the way you get the values using viper you will need to read to documentation to know which way works for you – Bergess

Answer 1

Viper doesn't have the ability to keep placeholders for values in key/value pairs, so I've managed to solve my issue with this code snippet:

log.Println("Loading config...")
viper.SetConfigName("application")
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
    panic("Couldn't load configuration, cannot start. Terminating. Error: " + err.Error())
}
log.Println("Config loaded successfully...")
log.Println("Getting environment variables...")
for _, k := range viper.AllKeys() {
    value := viper.GetString(k)
    if strings.HasPrefix(value, "${") && strings.HasSuffix(value, "}") {
        viper.Set(k, getEnvOrPanic(strings.TrimSuffix(strings.TrimPrefix(value,"${"), "}")))
    }
}

func getEnvOrPanic(env string) string {
    res := os.Getenv(env)
    if len(res) == 0 {
        panic("Mandatory env variable not found:" + env)
    }
    return res
}

This will overwrite all the placeholders found in the collection.

Photoemission

len(env) == 0 should be len(res) == 0 – Rubyeruch

Answer 2

Might not be the direct answer but the solution to the problem you're trying to solve.

Chances are you don't need to replace. If you leave the keys in yml empty and have viper.AutomaticEnv() on, then it will pick those values from environment variable. You'll need to add a replacer as well to match the keys with env names like:

viper.AutomaticEnv()
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
viper.SetConfigFile("config.yml")
config := &Config{}

Now, when you unmarhsal the config, you'll get the missing keys from env.

Example: RABBITMQ_PASSWORD env variable will be used to set rabbitmq.password if your yml looks like this:

rabbitmq
  password: ""

Accentual

this is the most underrated answer I've seen. It should solve issues for many developers using yamls. The linux env vars cannot have . in the name and with this we can replace dots with underscore in the env variables and still override the yaml config. – Jarlath

Answer 3

Update:

I extended the native yaml parser with this functionality and released it on github.

Usage:

type Config struct {
    Port     string   `yaml:"port"`
    RabbitMQ RabbitMQ `yaml:"rabbitmq"`
}

type RabbitMQ struct {
    Host     string `yaml:"host"`
    Port     string `yaml:"port"`
    Username string `yaml:"username"`
    Password string `yaml:"password"`
    Vhost    string `yaml:"vhost"`
}

func main() {
    var config Config
    file, err := ioutil.ReadFile("application.yaml")
    if err != nil {
        panic(err)
    }
    yaml.Unmarshal(file, &config)
    spew.Dump(config)
}

This is how application.yaml looks like:

port: ${SERVER_PORT}
rabbitmq:
  host: ${RMQ_HOST}
  port: ${RMQ_PORT}
  username: ${RMQ_USERNAME}
  password: ${RMQ_PASSWORD}
  vhost: test

vhost value will get parsed as usual, while everything surrounded with "${" and "}" will get replaced with environment variables.

Photoemission

Does it support default value? i.e: ${RMQ_HOST:dsa} – Brickey

It does not, if you need that feature we can arrange to add it into the repository with Backwards Compatibility. – Photoemission

Answer 4

You can explicitly substitute the env variables before calling the "Unmarshal" method. Assuming the configuration is stored in "Config" variable, the following code snippet should work.

log.Println("Loading config...")
viper.SetConfigName("application")
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
viper.AutomaticEnv()
if err := viper.ReadInConfig(); err != nil {
    fmt.Fprintf("Error reading config file %s\n", err)
}

for _, k := range viper.AllKeys() {
    v := viper.GetString(k)
    viper.Set(k, os.ExpandEnv(v))
}

if err := viper.Unmarshal(&Config); err != nil {
    fmt.Fprintf("Unable to decode into struct %s\n", err)
}

Ratfink

Answer 5

I resolved similar question by Using regexp to replace ENV firstly, here is my solutions:

# config.yaml
DB_URI: ${DB_USER}

and main.go:

package main

import (
    "fmt"
    "os"
    "regexp"

    "github.com/spf13/viper"
)

type DBCfg struct {
    DBURI string `mapstructure:"DB_URI"`
}

func main() {
    viper.SetConfigName("config")
    viper.SetConfigType("yaml")
    viper.AddConfigPath(".")
    viper.AutomaticEnv()

    if err := viper.ReadInConfig(); err != nil {
        panic(fmt.Errorf("Failed to read config"))
    }

    for _, key := range viper.AllKeys() {
        value := viper.GetString(key)
        envOrRaw := replaceEnvInConfig([]byte(value))
        viper.Set(key, string(envOrRaw))
    }

    var config DBCfg
    if err := viper.Unmarshal(&config); err != nil {
        panic(fmt.Errorf("failed to load"))
    }

    fmt.Println(config)
}

func replaceEnvInConfig(body []byte) []byte {
    search := regexp.MustCompile(`\$\{([^{}]+)\}`)
    replacedBody := search.ReplaceAllFunc(body, func(b []byte) []byte {
        group1 := search.ReplaceAllString(string(b), `$1`)

        envValue := os.Getenv(group1)
        if len(envValue) > 0 {
            return []byte(envValue)
        }
        return []byte("")
    })

    return replacedBody
}

and my output:

>>> DB_USER=iamddjsaio go run main.go

{iamddjsaio}

Popham

Not a common solution, doesn't work with structured config values such as map – Palpitant

Seems like the solution only resolve the env value item, structured config values should parse to your customize struct by tag when use viper, instead of use regexp. – Popham

Answer 6

I think an even better way to do this is to use Viper's concept of DecodeHooks when decoding configuration into a struct:

If you have a yaml configuration file like the following:

server:
  property: ${AN_ENV_VARIABLE}

Then inside the decode hook, grab the ENV variable as the decoding is happening:

decodeHook := func(f reflect.Type, t reflect.Type, data interface{}) (interface{}, error) {
    if f.Kind() == reflect.String {
        stringData := data.(string)
        if strings.HasPrefix(stringData, "${") && strings.HasSuffix(stringData, "}") {
            envVarValue := os.Getenv(strings.TrimPrefix(strings.TrimSuffix(stringData, "}"), "${"))
            if len(envVarValue) > 0 {
                return envVarValue, nil
            }
        }
    }
    return data, nil
}
err := viper.Unmarshal(c, viper.DecodeHook(decodeHook))

I think this could be further improved since you might have an incoming property that might need to be set to an int, for example. However, this general concept should work for most use cases.

Moidore