How to set dynamic values with Kubernetes yaml file

Question

For example, a deployment yaml file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
      spec:
        container:
          - name: guestbook
            image: {{Here want to read value from config file outside}}

There is a ConfigMap feature with Kubernetes, but that's also write the key/value to the yaml file. Is there a way to set the key to environment variables?

Ist

See also: https://mcmap.net/q/81286/-use-placeholders-in-yaml – Biosynthesis

Of all the options posted below, Kustomize is not the easiest tool to have variables in YAML files, but is the most native way to do this in kubernetes – since Kustomize is now directly usable inside kubectl. OP's exact use case would be the concept of Image Transformers but there's many other replacement options. The official documentation is VERY lacking in examples, so I always point people to this article from 2023 to learn the different Kustomize concepts: devopscube.com/kustomize-tutorial – Photofluorography

Answer 1

I don't think it is possible to set image through variable or Config Map in Kubernetes. But you can use for example Helm to make your deployments much more flexible and configurable.

Charkha

Answer 2

You can also use envsubst when deploying.

e.g.

cat app/deployment.yaml | envsubst | kubectl apply ...

It will replace all variables in the file with their values. We are successfully using this approach on our CI when deploying to multiple environments, also to inject the CI_TAG etc into the deployments.

Chortle

Also possible without cat: envsubst < deployment.yaml | kubectl apply -f - – Cozy

Do you have an example of envsubset and deployment so we can see syntax of the variables declarations and anchros? – Linebreeding

You just put $YOUR_ENV_NAME in the file, that's it. – Chortle

I put in a quick post to supplement this. Thanks. Ya - online I see all sorts of syntax $FOO ${FOO} {{FOO}} etc.. so it wasn't clear. envsubst < k8s/pls-invoice-rest-dep.yaml | kubectl apply -f - – Linebreeding

Edit queue is full, so I added here. cat app/deployment.yaml, there is no $. – Ginn

I agree with @NoamManos cat is so uncivilized !! :D – Heterochromous

Answer 3

You can't do it automatically, you need to use an external script to "compile" your template, or use helm as suggested by @Jakub.

You may want to use a custom bash script, maybe integrated with your CI pipeline.

Given a template yml file called deploy.yml.template very similar to the one you provided, you can use something like this:

#!/bin/bash

# sample value for your variables
MYVARVALUE="nginx:latest"

# read the yml template from a file and substitute the string 
# {{MYVARNAME}} with the value of the MYVARVALUE variable
template=`cat "deploy.yml.template" | sed "s/{{MYVARNAME}}/$MYVARVALUE/g"`

# apply the yml with the substituted value
echo "$template" | kubectl apply -f -

Goneness

Thank you for your clear answer! I have read helm document. It's a great tool. Your script is a right way to deploy an application with CI. – Ist

This was solution for me to template Kubernetes Job manifest. Thank you! – Rosenzweig

If I need to replace 2 variables how do I do that ? – Centimeter

Answer 4

One line:

cat app-deployment.yaml | sed "s/{{BITBUCKET_COMMIT}}/$BITBUCKET_COMMIT/g" | kubectl apply -f -

In yaml:

  ...
  containers:
  - name: ulisses
    image: niceuser/niceimage:{{BITBUCKET_COMMIT}}
  ...

Dendrology

or with the default value - cat app-deployment.yaml | sed "s/{{BITBUCKET_COMMIT}}/${BITBUCKET_COMMIT:=1}/g" | kubectl apply -f - – Hoeve

You all deserve the Useless Use of Cat Award! sed can read files perfectly fine on its own, all remaining arguments are parsed as input file paths. – Chessman

Thanks a lot, @Dendrology this is exactly what I was looking for. – Holmium

Is BITBUCKET_COMMIT an env variable? Where is the value actually set so that it can be substittuded? – Linebreeding

Hi... what if I need to replace 2 or 3 variable in the file – Centimeter

@yasinmohammed just add another sed entry in the command like cat ...| sed ...| sed ... | kubectl apply – Goneness

I added the sed and its working the kubectl apply part am not able to get cat filename | sed ... | sed ... | kubectl apply -f -n logs – Centimeter

I added the kubectl as a separate command after outputting to /tmp – Centimeter

Answer 5

After trying sed and envsubst I found Kustomize the most elegant and Kubernetes-native way. As an alternative also yq comes in handy sometimes.

Use Kustomize to change image name

Install the kustomize CLI (e.g. on a Mac this is brew install kustomize) and create a new file called kustomization.yaml in the same directory as your deployment.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

Now use the kustomize edit set image command to change the image name

# optionally define image name
IMAGE_NAME=ghcr.io/yourrepo/guestbook:c25a74c8f919a72e3f00928917dc4ab2944ab061

# replace image tag
kustomize edit set image $IMAGE_NAME

Finally apply your kustomized deployment.yml to your cluster using kubectl apply -k directory/where/your/kustomization/file/is like this:

kubectl apply -k .

For debugging you can see the resulting deployment.yml if you run kustomize build . :

$ kustomize build .
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
    spec:
      containers:
        - image: ghcr.io/yourrepo/guestbook:c25a74c8f919a72e3f00928917dc4ab2944ab061
          name: guestbook

Alternative: Use yq to change image name

Install the YAML processor yq (e.g. via homebrew brew install yq), define your variables and let yq do the replacement:

# define image name
IMAGE_NAME=ghcr.io/yourrepo/guestbook:c25a74c8f919a72e3f00928917dc4ab2944ab061

# replace image tag
yq e ".spec.template.spec.containers[0].image = \"$IMAGE_NAME\"" -i deployment.yaml

Now your deployment.yaml get's the new image version and then looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
    spec:
      containers:
        - image: ghcr.io/yourrepo/guestbook:c25a74c8f919a72e3f00928917dc4ab2944ab061
          name: guestbook

FYI: Your deployment.yaml isn't really valid Kubernetes configuration - the template.spec.container should not reside under the metadata tag - and also it is spelled containers.

Exotoxin

in kustomize, there is such concept called overlays, you can set different values for different environment; and i assume kustomize edit set <tag name> <tag value> does the same effect – Pupa

Answer 6

This kind of thing is painfully easy with ytt:

deployment.yml

#@ load("@ytt:data", "data")
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
      spec:
        container:
          - name: guestbook
            image: #@ data.values.image

values.yml

#@data/values
image: nginx@sha256:fe2fa7bb1ceb86c6d9c935bc25c3dd8cbd64f2e95ed5b894f93ae7ffbd1e92bb

Then...

$ ytt -f deployment.yml -f values.yml | kubectl apply -f -

or even better, use ytt's cousin, kapp for a high-control deployment experience:

$ ytt -f deployment.yml -f values.yml | kapp deploy -a guestbook -f -

Lambkin

is ytt standard on linux or something we need to install ahead of deploy? – Linebreeding

You install it: carvel.dev/ytt/docs/latest/install – Lambkin

Why are more people not talking about this, looks really great! – Valuer

@NicholasPetersen Yeah, it's unfortunate that we haven't been able to spread "the word" better about the Carvel tools like ytt. Glad you see value in it. Don't be shy in sharing with others you think would benefit. – Lambkin

Answer 7

I create a script called kubectl_create and use it to run the create command. It will substitute any value in the template that is referenced in an environment variable.

#!/bin/bash
set -e
eval "cat <<EOF
$(<$1)
EOF
" | kubectl create -f -

For example, if the template file has:

apiVersion: v1
kind: Service

metadata:
  name: nginx-external
  labels:
    app: nginx

spec:
  loadBalancerIP: ${PUBLIC_IP}
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 443

  selector:
    app: nginx

Run kubectl_create nginx-service.yaml and then the environment variable PUBLIC_IP will be substituted before running the actual kubectl create command.

Crystallize

why not out="$(cat $1)" and kubectl apply -f out ... maybe I am missing something – Gastongastralgia

Careful with shell eval and places where parameter substitution are a thing, as one can most likely do $( rm -rf / ) in the files. – Freedafreedman

Answer 8

yaml does not read values from another yaml file. As an alternative approach you could try this.

kind: Pod
metadata:
  creationTimestamp: null
  annotations:
    namespace: &namespaceId dev
    imageId: &imgageId nginx
    podName: &podName nginx-pod
    containerName: &containerName nginx-container
  name: *podName
  namespace: *namespaceId
spec:
  containers:
  - image: *imgageId
    name: *containerName
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Crim

How is this any better than just hardcoding these values? If they are in the same file... – Linebreeding

@AdamHughes it's better because there's a single source of truth for the values with this approach. This is important because say for example you forget to replace all, or there's a typo in one value. Instead of checking them all there's just this reference. Imagine a very long connection string, it would be tedious to maintain in more than one place in the file. – Roseanneroseate

Answer 9

My approach:

tools/jinja2-cli.py:

#!/usr/bin/env python3
import os
import sys
from jinja2 import Environment, FileSystemLoader

sys.stdout.write(Environment(loader=FileSystemLoader('templates/')).from_string(sys.stdin.read()).render(env=os.environ) + "\n")

Make rule:

_GENFILES = $(basename $(TEMPLATES))
GENFILES = $(_GENFILES:templates/%=%)

$(GENFILES): %: templates/%.j2 $(MKFILES) tools/jinja2-cli.py .env
        env $$(cat .env | xargs) tools/jinja2-cli.py < $< > $@ || (rm -f $@; false)

Inside the .j2 template file you can use any jinja syntax construct, e.g. {{env.GUEST}} will be replaced by the value of GUEST defined in .env

So your templates/deploy.yaml.j2 would look like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
      spec:
        container:
          - name: guestbook
            image: {{env.GUEST}}

Another approach (using just bash builtins and xargs) might be

env $(cat .env | xargs) cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: guestbook
      spec:
        container:
          - name: guestbook
            image: ${GUEST}
EOF

Gewgaw

Answer 10

I think the standard - Helm should be used instead of custom scripts to solve this problem nowadays. You don't need to deploy to generate Kubernets yamls on the machine.

An example:

1. Install helm on your machine so helm command exists

2. https://artifacthub.io/packages/helm/pauls-helm-charts/helloworld - Install button

3. helm repo add pauls-helm-charts http://tech.paulcz.net/charts

4. helm pull pauls-helm-charts/helloworld --version 2.0.0

5. tar -zxvf helloworld-2.0.0.tgz && cd helloworld

6. helm template -f values.yaml --output-dir helloworld . --namespace my-namespace --name-template=my-name

So it created these files from values.yaml:

wrote helloworld/helloworld/templates/serviceaccount.yaml
wrote helloworld/helloworld/templates/service.yaml
wrote helloworld/helloworld/templates/deployment.yaml

Inside values.yaml, you can change predefined repository (or 100% any value can be repeated in Kubernetes yamls as you wish):

image:
  repository: paulczar/spring-helloworld

Now if you want to deploy, make sure kubectl works and just apply these generated files using kubectl apply -f serviceaccount.yaml, etc.

Moldboard

Answer 11

I have been using kubetpl

It has three different template flavors and supports ConfigMap/Secret freezing.

Overplay

Answer 12

If you just want to change the image or a tag while your deployment is running, you could set the image of a specific container in your deployment:

kubectl apply -f k8s
kubectl set image deployments/worker-deployment worker=IMAGE:TAG

Cretic

Answer 13

I create a script called kubectl_apply. It loads variables from .env, replace ${CUSTOMVAR} in yml and pass it to kubectl command

  #!/bin/bash
  set -a
  source .env
  set +a
  eval "cat <<EOF
  $(<$1)
  EOF
  " | kubectl apply -f -

Riocard

Answer 14

I've published a command-line tool ysed that helps exactly with that, in case you plan to script it.

Essary

Answer 15

create a file called kubectl_advance as below and enjoy calling it just like kubectl commands.

e.g.

EXPORT MY_VAL="my-v1"

kubectl_advance -c -f sample.yaml # -c option is to call create command
kubectl_advance -r -f sample2.yaml # -r option is to call replace command

Assuming the yaml file has the value like ${MY_VAL} to be replaced by the environment variable.

#!/usr/bin/env bash

helpFunction()
{
   echo "Supported option is [-f] for file"
   exit 1
}

while getopts "f:cr" opt
do
   case "$opt" in
      f ) yamlFile="$OPTARG" ;;
      c ) COMMAND_IS_CREATE="true" ;;
      r ) COMMAND_IS_REPLACE="true" ;;
      ? ) helpFunction ;; # Print helpFunction in case parameter is non-existent
   esac
done

echo 'yaml file is : '$yamlFile

YAML_CONTENT=`eval "cat <<EOF
$(<$yamlFile)
EOF
"`

echo 'Final File Content is :=>'
echo '------------------'

echo "$YAML_CONTENT"


if [[ "$COMMAND_IS_CREATE" == "true" ]]; then
  COMMAND="create"
fi

if [[ "$COMMAND_IS_REPLACE" == "true" ]]; then
  COMMAND="replace"
fi

echo "$YAML_CONTENT" | kubectl $COMMAND -f -

Translocation

Answer 16

Per YAML's spec, there's no way to do that. but...

This is possible with yq's explode operator.

Namespace, Deployment, ClusterIP, Ingress... Clean and DRY:

cat <<EOF | yq 'explode(.)' | kubectl apply -f -
kind: Namespace
apiVersion: v1
metadata:
  name: &namespace example-namespace
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: example-deployment
  namespace: *namespace
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: &app-label example-app
    spec:
      containers:
      - name: httpecho
        image: hashicorp/http-echo:0.2.3
        args:
        - "-listen=:5678"
        - "-text=hello world 1"
        ports:
        - containerPort: &app-port 5678
  selector:
    matchLabels:
      app: *app-label
---
kind: Service
apiVersion: v1
metadata:
  name: &app-cluster-ip example-service
  namespace: *namespace
spec:
  type: ClusterIP
  selector:
    app: *app-label
  ports:
  - name: http
    protocol: TCP
    port: *app-port
    targetPort: *app-port
---
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: example-ingress
  namespace: *namespace
spec:
  ingressClassName: caddy
  rules:
  - host: example.kubernetes.localhost
    http:
      paths:
      - path: /hello
        pathType: Prefix
        backend:
          service:
            name: *app-cluster-ip
            port:
              number: *app-port
EOF

Barilla

Answer 17

In OKD (a superset of k8s) Template objects (docs) are parametrized - see the parameters section below. You can put any number of objects inside Templates such as Deployments/DeploymentConfigs, Services, Ingresses/Routes, etc.

Note these parameters support not just strings but also numerical values (see the double curly braces around REPLICA_COUNT below) and even random number generators for auto-generating fresh application credentials at each container startup (see GITHUB_WEBHOOK_SECRET below):

kind: Template
apiVersion: template.openshift.io/v1
metadata:
  name: my-template
objects:
  - kind: BuildConfig
    apiVersion: build.openshift.io/v1
    metadata:
      name: cakephp-mysql-example
      annotations:
        description: Defines how to build the application
    spec:
      source:
        type: Git
        git:
          uri: "${SOURCE_REPOSITORY_URL}" 
          ref: "${SOURCE_REPOSITORY_REF}"
        contextDir: "${CONTEXT_DIR}"
  - kind: DeploymentConfig
    apiVersion: apps.openshift.io/v1
    metadata:
      name: frontend
    spec:
      replicas: "${{REPLICA_COUNT}}" 
parameters:
  - name: SOURCE_REPOSITORY_URL 
    displayName: Source Repository URL 
    description: The URL of the repository with your application source code 
    value: https://github.com/sclorg/cakephp-ex.git 
    required: true 
  - name: GITHUB_WEBHOOK_SECRET
    description: A secret string used to configure the GitHub webhook
    generate: expression 
    from: "[a-zA-Z0-9]{40}" 
  - name: REPLICA_COUNT
    description: Number of replicas to run
    value: "2"
    required: true
message: "... The GitHub webhook secret is ${GITHUB_WEBHOOK_SECRET} ..." 

Rode

Answer 18

Helm is exactly meant for such things and a lot more. It handle complex set of resource deployment as a group etc.

But if we are still looking for some simple alternative then how about using ant?

If you want to modify the file as part of build process or test process then you can go with ant task as well.

Using ant you can load all environment values as property or you can simply load properties file like:

<property environment="env" />
<property file="build.properties" />

Then you can have a target which converts template files into your desired yaml file.

<target name="generate_from_template">

    <!-- Copy task to replaces values and create new file -->
    <copy todir="${dest.dir}" verbose="true" overwrite="true" failonerror="true">

        <!-- List of files to be processed -->
        <fileset file="${source.dir}/xyz.template.yml" />

        <!-- Mapper to transform filename. Removes '.template' from the file
            name when copying the file to output directory -->
        <mapper type="regexp" from="(.*).template(.*)" to="\1\2" />

        <!-- Filter chain that replaces the template values with actual values 
            fetched from properties file -->
        <filterchain>
            <expandproperties />
        </filterchain>
    </copy>
</target>

Of course you can use a fileset instead of file in case you want to change values dynamically for multiple files (nested or whatever)

Your template file xyz.template.yml should look like:

apiVersion: v1
kind: Service
metadata:
  name: ${XYZ_RES_NAME}-ser
  labels:
    app: ${XYZ_RES_NAME}
    version: v1
spec:
  type: NodePort
  ports:
    - port: ${env.XYZ_RES_PORT}
      protocol: TCP
  selector:
    app: ${XYZ_RES_NAME}
    version: v1

env. property being loaded from environment variables and other from property file

Hope it helped :)

Dhiren

Answer 19

In the jitsi project the tpl == frep command is used to substitute values, an extension to envsubst

https://github.com/jitsi/docker-jitsi-meet/issues/65

I keep on using the old shell tools like sed and friends but such code is quickly unreadable when its more than a handful of value to deal with.

Inherit

Answer 20

For my deployments, I typically use Helm charts. It requires me to update values.yaml files periodically.

For dynamically updating YAML files, I used 'envsubst' since it is simple and does not require sophisticated configuration. In addition, most of the tools only work with valid Kubernetes manifests, not simple YAML files.

I created a simple script to handle the YAML modification to simplify the usage

https://github.com/alexusarov/vars_replacer

Example:

./vars_replacer.sh -i [input_file] -o [output_file] -p "[key=value] [key=value]"

Bawbee