UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Question

Yesterday, I stopped a Helm upgrade when it was running on a release pipeline in Azure DevOps and the followings deployments failed.

I tried to see the chart that has failed with the aim of delete it but the chart of the microservice ("auth") doesn't appear. I used the command «helm list -n [namespace_of_AKS]» and it doesn't appear.

What can I do to solve this problem?

Error in Azure Release Pipeline

2022-03-24T08:01:39.2649230Z Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
2022-03-24T08:01:39.2701686Z ##[error]Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Helm List

Transcendentalistic

helm ls -a -n {namespace} will list all releases within a namespace, regardless of status. You can also use helm ls -aA instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before) – Woven

@BlenderFox thank you, your response solved my problem – Transcendentalistic

@kavyasaraboju-MT thank you for your response. I didn't prove it because I coul fix with the previous response – Transcendentalistic

@Transcendentalistic no problem. I will add as an answer for others to find – Woven

Answer 1

This error can happen for few reasons, but it most commonly occurs when there is an interruption during the upgrade/install process as you already mentioned.

To fix this one may need to, first rollback to another version, then reinstall or helm upgrade again.

Try below command to list

helm ls --namespace <namespace>

but you may note that when running that command ,it may not show any columns with information

Try to check the history of the previous deployment

helm history <release> --namespace <namespace>

This provides with information mostly like the original installation was never completed successfully and is pending state something like STATUS: pending-upgrade state.

To escape from this state, use the rollback command:

helm rollback <release> <revision> --namespace <namespace>

revision is optional, but you should try to provide it.

You may then try to issue your original command again to upgrade or reinstall.

Decoy

To rollback to the previous revision and you don't know the number of the revision, you can use helm rollback <release> 0 and it will do it automatically. – Cartwheel

Do you know how can I reproduce this behavior, in oder to test a automation? – Buyer

I use TrueNAS and rolled back to a prior version. After trying the upgrade again, it worked this time. – Masaryk

To clarify something that confused me for a second : <release> is something from the NAME column returned by helm ls – Redeploy

I have a similar problem here. My helm upgrade fails with this same error the first time it runs, but the retry is successful. When I do a helm list my last deployment shows as successful. – Doddering

Answer 2

helm ls -a -n {namespace} will list all releases within a namespace, regardless of status.

You can also use helm ls -aA instead to list all releases in all namespaces -- in case you actually deployed the release to a different namespace (I've done that before)

Woven

to delete: helm uninstall {release} -n {namespace} – Prowess

Answer 3

Try deleting the latest helm secret for the deployment and re-run your helm apply command.

kubectl get secret -A | grep <app-name>
kubectl delete secret <secret> -n <namespace>

Arnett

I had this problem when a deployment failed due to a repository name clash. I was not able to use rollback. but deleting the secret did it for me! Thanks! – Vulgarian

you saved me from a lot of headache! deleting the secret worked a charm – Stifling

to get only latest you can just add | tail -n 1 in the end of the first command so you will have: kubectl get secret -A | grep <app-name> | tail -n 1 – Wendling

Answer 4

I also got the same issue, I solved it by rolling back the Helm release from the OpenLens tool, It was showing as Pending-Upgrade before.

Crenel

Answer 5

My exact error was

Helm upgrade failed: another operation (install/upgrade/rollback) is in progress" or "##[error]Error: UPGRADE FAILED: release <namespace> failed, and has been rolled back due to atomic being set: timed out waiting for the condition

The error has 2 potential solutions: Either delete the stuck deployment, or fix something in deployment pipeline.

Solution #1: Delete the stuck deployment

If this same pipeline has been working until now, it may just be stuck. Time to delete it in the Azure portal!

1. Go the Azure Portal > Azure CLI (aka the icon called "Cloud Shell")

2. az account show // This will give which subscription are you in.

3. az account set --subscription

4. az aks get-credentials --name --resource-group

5. kubectl get secrets -n

6. In this situation, the latest release has a problem, so delete that release by running the following:

7. kubectl delete secret -n sh.helm.release.v1..v88 (or whatever the last release was as shown after last command)

8. Run the pipeline again, it works :-)

Solution #2: Fix the pipeline

If you've just updated your pipeline, or haven't run it yet, you may have a bug. That was the case for my particular Azure DevOps pipeline.

1. The azure-pipeline.yml stage is missing some parameters (or whatever the problem is in your pipeline)
2. Fix it.
3. Deployment succeeds!

Distributary

Answer 6

Delete the helm secret associated with the release and re-run the command

Helm3 makes use of the Kubernetes Secrets object to store any information regarding a release. These secrets are basically used by Helm to store and read it's state every time we run "helm upgrade" or "helm install".

First we get all the secrets for the namespace by running: "kubectl get secrets -n <<namespace"

Now delete the helm secret associated with the pending release by running the following command:

Note: Here "gloo-system" is the namespace.

Rennie

I have a similar problem which I asked here. My latest deployment was successful but every time I upgrade a new release, it fails first. Once I retry its successful then. How can I find more on whats going on in the secrets? – Doddering