Kubernetes Helm stuck with an update in progress

Question

I tried to run a Helm upgrade before running helm repo update and now it seems to be permanently stuck in "STATUS: pending-upgrade" and won't let me try to run the upgrade again.

Trying to run: helm upgrade --namespace coder --install --force --atomic --wait --version 1.13.2 --values ./coder.yaml coder coder/coder

outputs: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Kavita

Answer 1

This solution worked for me:

kubectl get secrets
kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

Following the resolution described in this issue

Disengage

This answer explains why, but this one solved my problem. I've canceled a pipeline run in the middle of an upgrade command and there was no Release registered in helm to do rollback or uninstall. Deleting the helm's release secret was the only way to move on. +1 – Robichaux

just to add - I had to add --namespace <name-of-namespace> to both kubectl commands to get them to work for my deployment (otherwise it assumes default namespace) – Genro

this worked for me also, in the secret labels you should see "status : pending" – Univalent

This is probably the worst advice. You WILL most probably end up with dangling resources. A rollback is by far the safest way to restore to previous state. If it is the first helm install (i.e. no release versions), just force delete. – Hillary

i did not see any helm releases other than "deployed" status with helm ls -A but still received the "an operation is running" message. After deleting the latest helm secret i could upgrade again. So: worked for me. – Hastie

Answer 2

TLDR: You need to rollback to another version first and then helm upgrade again:

helm rollback <release> <revision> --namespace <namespace>

---

This can happen for a few reasons, but it ultimately occurs when there's an interruption during the upgrade/install process. Commonly, you SIGKILL (Ctrl C) while the deployment is ongoing.

You'll notice that if you helm ls --namespace <namespace> while it's stuck in STATUS: pending-upgrade state, you'll see the following without any other information:

NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION

The best workaround currently is to rollback to another version, and then helm upgrade again:

helm rollback <release> <revision> --namespace <namespace>

revision is optional, but you should try to provide it.

more resources:

• https://github.com/helm/helm/issues/8987
• https://github.com/helm/helm/issues/4558

---

EDIT

If this is your first revision, use:

helm delete <release> -n <namespace>

Isoprene

But what do you do if this is the first Revision? There is nothing to roll back to. – Thriller

i also had this issue but using flux, and this solution worked for that as well – Asynchronism

to get the stuck app listet by helm use helm ls --namespace <namespace> -all – Apennines

@Thriller just helm delete <release> (if there is no previous revision) – Hillary

if you have many deployments, you can run the following command to rollback all the failing deployments helm ls -a -n <namespace> | grep -e failed -e pending-upgrade | awk '{print $1} {print $2} {print $3 - 1}' | xargs -n 3 /bin/sh -c 'helm rollback $0 $2 -n $1' – Dessertspoon

I want to add: This also works if you have a pending-rollback situation. In my case, I mistakenly killed the rollback job, leading to a forever pending-rollback state. Manually rolling back worked and allowed me to continue with my next deployment. – Gaillardia

Answer 3

In case is useful to someone, and in response to explicitsoul's comment, what fixed it to me was just:

helm delete <release> -n <namespace>

That removed the pending install (in my case, the first one so I hadn't a previous release to rollback to) and then I was able to run the install again.

What caused the stuck process in my case was a CTRL-C canceling the install command, so don't do that.

Swarthy

Answer 4

Here is what worked for me

1. helm list --all This will list all the releases with their status

NAME  NAMESPACE       REVISION        UPDATED                                 STATUS               CHART                   APP VERSION
rel1  default         1               2021-06-04 14:15:37.652066 +0530 IST    deployed             rel1-3.32.0             0.46.0     
rel2  default         29              2021-06-18 11:02:38.779801 +0530 IST    pending-upgrade      rel2-0.0.1                     
rel3  default         3               2021-06-17 11:27:14.608042 +0530 IST    deployed             rel3-0.0.1      

2. Notice that rel2 has status as pending-update . This happened because I did a Ctrl+C while upgrade was in progress
3. All i had to do was rollback to previous revision in this case 28 helm rollback rel2 28 --namespace default

NAME   NAMESPACE       REVISION        UPDATED                                 STATUS          CHART          APP VERSION
rel1   default         1               2021-06-04 14:15:37.652066 +0530 IST    deployed        rel1-3.32.0    0.46.0     
rel2   default         30              2021-06-18 11:26:07.555547 +0530 IST    deployed        rel2-0.0.1                     
rel3   default         3               2021-06-17 11:27:14.608042 +0530 IST    deployed        rel3-0.0.1     

Deceptive

Answer 5

I am facing the same. I used helm 3.4.1... It happens when the deployment is in pending and you use --atomic (which in helm3 implies also --wait).

I could not get upgrade working. Worst is that even helm -n code list did not show anything, so I could not do:

helm -n code code 

As helm3 holds such info in secrets, just clean the respective secret(s) and do install (or upgrade --install , but without --atomic). In your case something like

helm delete --namespace code secret sh.helm.release.v1.code.v1

(where last v1 is the release number, so maybe list and delete all if you are ok with that).

and afterwars helm install.

NOTE: old objects (pods,etc) will be there, so the new install will try to merge things. It went OK for me, but note -> It's a HACK :)

More on: https://github.com/helm/helm/issues/5595

Sickness

Answer 6

In order to rollback to previous version you can just pass the release name:

helm rollback <RELEASE_NAME>

Where RELEASE_NAME can be seen when you run helm list --all -> under the NAME column.

(*) Add --namespace <namespace> if terminal context is not set to namespace.

Coronel

Answer 7

These are the steps that worked for me:

1. See the status of your deployment (my was pending all the time)

   helm list --all

2. Rollback to the previous version, for me was working already here, next step optional

   helm rollback <NAMESPACE_NAME> <Previous Version> --namespace <NAMESPACE_NAME>

3. In case you want new/another deploy

   helm upgrade . . .

Paracelsus

Shouldn't it be helm rollback <RELEASE_NAME> ... instead of <NAMESPACE_NAME>? – Assyriology

Answer 8

kubectl get secrets kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

By using the above command, it will remove the existing secrets in the middle of helm upgrade which will run and delete the stucked helm upgrade and it will generate the new one to proceed with the new upgrade.

Touchline

Answer 9

Although you can delete the secret to fix your problem (as mentioned in one of the solutions above), the best way is to do it with helm rollback (as also mentioned in a solution above). The helm rollback solution not only unblocks your path to make a new deployment (upgrade) but also brings your resources back to their previous working state.

Similarly, if the helm install failed on the first release and there is no working release to rollback, it's better to helm uninstall (previously was helm delete) rather than deleting the secret.

Excite