How not to overwrite randomly generated secrets in Helm templates

Question

I want to generate a password in a Helm template, this is easy to do using the randAlphaNum function. However the password will be changed when the release is upgraded. Is there a way to check if a password was previously generated and then use the existing value? Something like this:

apiVersion: v1
kind: Secret
metadata:
  name: db-details
data:
  {{ if .Secrets.db-details.db-password }}
  db-password:  {{ .Secrets.db-details.db-password | b64enc }}
  {{ else }}
  db-password: {{ randAlphaNum 20 | b64enc }}
  {{ end }}

Unworldly

Generate & pass secret from outside – Buckjumper

Answer 1

You can build on shaunc's idea to use the lookup function to fix the original poster's code like this:

apiVersion: v1
kind: Secret
metadata:
  name: db-details
data:
  {{- if .Release.IsInstall }}
  db-password: {{ randAlphaNum 20 | b64enc }}
  {{ else }}
  # `index` function is necessary because the property name contains a dash.
  # Otherwise (...).data.db_password would have worked too.
  db-password:  {{ index (lookup "v1" "Secret" .Release.Namespace "db-details").data "db-password" }}
  {{ end }}

Only creating the Secret when it doesn't yet exist won't work because Helm will delete objects that are no longer defined during the upgrade.

Using an annotation to keep the object around has the disadvantage that it will not be deleted when you delete the release with helm delete ....

Labbe

While I do like this approach, it doesn't seem to work well with helm lint, helm template and helm --dry-run because it doesn't (and shouldn't) contact the Kubernetes API server to fetch the existing secret. The docs state that "the lookup function will return an empty list (i.e. dict) in such a case". This results in "nil pointer evaluating interface" (when called without index) or "error calling index: index of untyped nil" when called with index. Ideas? – Subvene

@Subvene I guess under helm lint all the fields in .Release are not setup, so the simple fix would be to replace if .Release.IsInstall with if not .Release.IsUpdate (well, avoid the negation and swap the two branches, but you get my point). This change should also help with helm template. – Labbe

Sorry, can no longer edit the comment; it should be if not .Release.IsUpgrade – Labbe

I went with an assignment and a conditional. {{- $data := (lookup ...).data -}} and then {{- if $data }}. On another note: It seems to be a different story for ConfigMaps. When keys are entirely omitted in a Helm upgrade run, but present in the existing ConfigMap object on the cluster, they remain unchanged. – Subvene

Answer 2

I've got a lot of trouble with the answers from Jan Dubois and shaunc. So I built a combined solution.

The downside of Jan's answer: It leads to errors, when it is used with --dry-run.
The downside of shaunc's answer: It won't work, because the resources will be deleted on helm upgrade.

Here is my code:

# store the secret-name as var
# in my case, the name was very long and containing a lot of fields
# so it helps me a lot
{{- $secret_name := "your-secret-name" -}}

apiVersion: v1
kind: Secret
metadata:
  name: {{ $secret_name }}

data:
  # try to get the old secret
  # keep in mind, that a dry-run only returns an empty map 
  {{- $old_sec := lookup "v1" "Secret" .Release.Namespace $secret_name }}

  # check, if a secret is already set
  {{- if or (not $old_sec) (not $old_sec.data) }}
  # if not set, then generate a new password
  db-password: {{ randAlphaNum 20 | b64enc }}
  {{ else }}
  # if set, then use the old value
  db-password: {{ index $old_sec.data "db-password" }}
  {{ end }}

Cornstalk

It's a shame it doesn't work with upgrade dry-run. It makes it look like it's going to rewrite the secret but then it doesn't. – Dianthe

@Dianthe There's no way it will ever work with upgrade --dry-run (where the lookup returns an empty dict) and randomly generated password. Should the generated password be a hash of some stable data relevant just for the release (e.g. release name, namespace), it could work. With the release name itself being randomly generated this should even be safe enough to use. – Murillo

You need to use --dry-run=server to account for the lookup. – Haar

Answer 3

It's still one of the biggest issues of Helm. As far as I understand no good solution is available yet (see https://github.com/helm/charts/issues/5167).

One dirty workaround is to create secret as pre-install hook. Obvious downside of this approach is that secret will not be deleted on helm delete.

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "helm-random-secret.fullname" . }}
  annotations:
    "helm.sh/hook": "pre-install"
    "helm.sh/hook-delete-policy": "before-hook-creation"
  labels:
    app: {{ template "helm-random-secret.name" . }}
    chart: {{ template "helm-random-secret.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  some-password: {{ default (randAlphaNum 10) .Values.somePassword | b64enc | quote }}

Benson

I have a doubt about pre-install hook. What if you add new chart to requirements, that required new secret? You are using helm upgrade then, not helm install... It looks for me there is simply no reasonable way to use generated passwords with helm (which makes me wonder, why they have added functions for generating random passwords in first line). – Woodring

Helm supports pre-upgrade hooks as well. – Capitoline

Yeah this works as a charm already for few years – Yonne

This is how Sentry's Helm charts work: github.com/sentry-kubernetes/charts/blob/develop/sentry/… – Rumormonger

Answer 4

You can use the lookup function and skip generation if secret already exists:

{{- if not (lookup "v1" "secret" .Release.Namespace "db-details") -}}
<create secret here>
{{- end -}}

Limoli

I like this idea. Is there a reason to use lookup instead of Release.IsUpgrade ? – Pekan

Hmm... well -- suppose the upgrade actually adds the secret? You could work around that but perhaps this is simpler. – Limoli

Actually, this method doesn't work as intended. As @jan-dubois correctly pointed out in his answer below, the secret is deleted on upgrade if it already exists. – Pekan

Answer 5

Building on shaunc's idea to use the lookup function, I've created the following template:

{{/*
Returns a secret if it already in Kubernetes, otherwise it creates
it randomly.
*/}}
{{- define "getOrGeneratePass" }}
{{- $len := (default 16 .Length) | int -}}
{{- $obj := (lookup "v1" .Kind .Namespace .Name).data -}}
{{- if $obj }}
{{- index $obj .Key -}}
{{- else if (eq (lower .Kind) "secret") -}}
{{- randAlphaNum $len | b64enc -}}
{{- else -}}
{{- randAlphaNum $len -}}
{{- end -}}
{{- end }}

Then you can simply configure secrets like:

---
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  PASSWORD: "{{ include "getOrGeneratePass" (dict "Namespace" .Release.Namespace "Kind" "Secret" "Name" "my-secret" "Key" "PASSWORD") }}"

Undergrowth

Answer 6

The actual tools are all here. My workaround is just another combination of suggested tools

{{- if not (lookup "v1" "Secret" .Release.Namespace "mysecret") }}
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    "helm.sh/resource-policy": "keep"
type: Opaque
stringData:
  password: {{ randAlphaNum 24 }}
{{- end }}

So if there is no such secret, it will be created. If the secret is present, it will be removed from the chart, but not from the cluster, the "helm.sh/resource-policy": "keep" will prevent it.

You may ask (as someone already did above) why lookup, not .Release.IsUpdate. Imagine the situation: your secret is a password to a database. You keep the data in the persistent volume, the claim for which is also annotated by "helm.sh/resource-policy": "keep", so if you even uninstall and reinstall the chart, the data would persist. If you do so with .Release.IsUpdate as condition, then you password will be recreated, the old password will be lost and you will loose the access to your data. If you query for the secret existence, it won't happen.

Gabie

Answer 7

I've rewritten kubernetes replicator and added some annotations to deal with this kind of problems: https://github.com/olli-ai/k8s-replicator#use-random-password-generated-by-an-helm-chart

Now can generate a random password with helm, and replicate it only once to another secret thus it won't be change by helm in the future.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: admin-password-source
  annotations:
    k8s-replicator/replicate-to: "admin-password"
    k8s-replicator/replicate-once: "true"
stringData:
  password: {{ randAlphaNum 64 | quote }}

Hope it will help people.

Anatol

Answer 8

You can leverage definitions in the _helpers.tpl

_helpers.tpl

{{/*
Create the secret name
*/}}
{{- define "mssql-server.secretName" -}}
{{- include "mssql-server.name" . }}-mssql-secret
{{- end }}

{{/*
Get sa password value
*/}}
{{- define "mssql-server.sapassword" -}}
{{- if .Release.IsInstall -}}
{{ .Values.sa_password | default (randAlphaNum 20) | b64enc | quote }}
{{- else -}}
{{ index (lookup "v1" "Secret" .Release.Namespace (include "mssql-server.secretName" .)).data "sa_password" }}
{{- end }}
{{- end }}

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ include "mssql-server.secretName" . }}
  labels:
    {{- include "mssql-server.labels" . | nindent 4 }}
type: Opaque
data:
  sa_password: {{ include "mssql-server.sapassword" . }}

Monamonachal

Answer 9

you can use Immutable Secrets https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-database-password
  namespace: {{ .Release.Namespace }}
  labels:
    app: {{ .Release.Name }}-database
  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-delete-policy": before-hook-creation
type: Opaque
immutable: true
data:
  db-password: {{ randAlphaNum 20 | b64enc | quote }}

Doormat

Answer 10

A bit late here, and most people may just catch it in the documentation:

helm does this for you with the annotation "helm.sh/resource-policy": keep

see:

https://helm.sh/docs/howto/charts_tips_and_tricks/#tell-helm-not-to-uninstall-a-resource

Gnathous