Recreate job or delete old job on every helm install

Question

I have created Kubernetes job and job got created, but it is failing on deployment to the Kubernetes cluster. When I try to redeploy it using Helm the job is not redeploying (deleting old job and recreating new one, unlike a microservice deployment).

How can I achieve this redeploying job without manually deleting it in Kubernetes? Could I tell it to recreate the container?

job.yaml contains:

apiVersion: batch/v1
kind: Job
metadata:
  name: "{{ .Release.Name }}-init-job"
  namespace: {{ .Release.Namespace }}
spec:
  template:
    metadata:
      annotations:
        linkerd.io/inject: disabled
        "helm.sh/hook-delete-policy": before-hook-creation
        "helm.sh/hook": pre-install,pre-upgrade,pre-delete
        "helm.sh/hook-weight": "-5"
    spec:
      serviceAccountName: {{ .Release.Name }}-init-service-account
      containers:
        - name: app-installer
          image: some image
          command:
            - /bin/bash
            - -c
            - echo Hello executing k8s init-container
          securityContext:
            readOnlyRootFilesystem: true
      restartPolicy: OnFailure

Job is not getting redeployed

kubectl get jobs -n namespace

NAME                    COMPLETIONS   DURATION   AGE
test-init-job   0/1           13h        13h

kubectl describe job test-init-job -n test

Name:           test-init-job
Namespace:      test
Selector:       controller-uid=86370470-0964-42d5-a9c1-00c8a462239f
Labels:         app.kubernetes.io/managed-by=Helm
Annotations:    meta.helm.sh/release-name: test
                meta.helm.sh/release-namespace: test
Parallelism:    1
Completions:    1
Start Time:     Fri, 14 Oct 2022 18:03:31 +0530
Pods Statuses:  0 Running / 0 Succeeded / 1 Failed
Pod Template:
  Labels:           controller-uid=86370470-0964-42d5-a9c1-00c8a462239f
                    job-name=test-init-job
  Annotations:      helm.sh/hook: pre-install,pre-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                    helm.sh/hook-weight: -5
                    linkerd.io/inject: disabled
  Service Account:  test-init-service-account
  Containers:
   test-app-installer:
    Image:      artifactory/test-init-container:1.0.0
    Port:       <none>
    Host Port:  <none>
    Environment:
      test.baseUrl:         baser
      test.authType:        basic
      test.basic.username:  jack
      test.basic.password:  password
    Mounts:
      /etc/test/config from test-installer-config-vol (ro)
  Volumes:
   test-installer-config-vol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      test-installer-config-files
    Optional:  false
Events:        <none>

Pacien

When you redeploy it, has anything changed? The general model for a Job is that it will run exactly once, and IME it can cause problems when you try to manage it in Helm. Does adding a Helm hook annotation make it run in the way you expect? (I can expand further on that, if it's the behavior you want.) – Refractive

I have updated yaml file i tried with hooks . I want job to recreated/redeployed every time when I do helm install. but job is not been deleted automatically and it throws exception. I need to delete it manually to redeploy the job . – Pacien

With the Helm hook annotations you've added in I would expect every helm upgrade command to delete and recreate the Job. It's possible you'll need to delete it once more before running the first upgrade. – Refractive

kubectl get jobs -n namespace NAME COMPLETIONS DURATION AGE test-init-job 0/1 13h 13h – Pacien

it is not getting redeployed – Pacien

Answer 1

Jobs cannot be redeployed, because their template is immutable.

If you get a message cannot patch "..." with kind Job: Job.batch "db-sync-download-snapshot" is invalid: spec.template: Invalid value: ... field is immutable, this is the reason.

Just delete a job, or create it with a $VERSION appended to a name.

With GitOps, $VERSION should be $CI_COMMIT_HASH.

Replace the name within the Helm template:

  metadata:
    name: "{{ .Release.Name }}-init-job-{{ .Release.revision }}"

Governance

Answer 2

set ttlSecondsAfterFinished, see: https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs

Eran

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Brunel