Helm Release stuck in Uninstalling state

Question

I was trying to uninstall a helm release in my AKS cluster using helm uninstall RELEASE_NAME but seems like it failed. The failure happened because for some reason all of the nodes in my cluster went in a not ready state when I used helm uninstall.

Since then, I got all the nodes to get back up and running and the cluster is functioning as it should. Now, when I do try helm list, I don't get to see this release present anymore, but doing a helm list -a shows me that the state of the release is still in uninstalling state. I have tried quite a few things, but it has been stuck there now for the last 3-4 days.

A few things that I tried was to use helm uninstall RELEASE_NAME, helm delete --purge RELEASE_NAME but these commands throw an error because the release is already in uninstalling state. I tried modifying the helm secrets in the cluster for this specific release but that didn't help either. Helm3 is being used so it is not like I can restart the tiller pod to maybe stabilize this.

Does Helm3 use some kind of a finalizer mechanism which can be modified to rectify this or is there no alternate way in which I can perhaps try to delete this release? I want to use the same release name for the specific API in the future too.

Any help will be really appreciated.

Ashcroft

helm hist releasename , helm rollback releasename versionnumber-with-status-deployed. you will be good to delete after rollback – Plasmosome

@Adiii, I tried to do this as well, but now the status shows up as pending rollback. I guess because it is stuck in the uninstalling state, so once that is resolved only then some other operations will work on the same release. – Ashcroft

did you tried to delete secret for the current stuck release? k delete secrets sh.helm.release.v1.name.v4 – Plasmosome

Ok, I tried modifying the secrets but have not tried deleting them altogether. Will deleting it work? Anyway, I'll surely give it a try! – Ashcroft

yes, but make sure to delete the version that is being stuck – Plasmosome

Ok, so @Adiii, sh.helm.release.v1.name.v1 shows the status as uninstalling while sh.helm.release.v1.name.v2 shows the status as pending rollback. Should I be deleting both the secrets or the one with the uninstalling status that is v1? – Ashcroft

Let us continue this discussion in chat. – Plasmosome

Answer 1

Based on the discussion, the following steps resolve the issue.

helm hist releasename
helm rollback releasename versionnumber-with-status-deployed

if this did not help, then delete secret for each version

helm hist releasename
kubectl get secrets
k delete secrets sh.helm.release.v1.name.VERSION-N

Plasmosome

I found the kubectl delete secrets hack to be necessary after failing to uninstall a release of an old chart referring to now-gone CRDs. – Frankly

FWIW, I had a release I physically couldn't rollback, since it was the 1st one. Deleting the secret worked perfectly. – Griner

Thank-you. Similarly I had to delete all helm.sh/release.v1 secrets (not just the latest), after which all was good. To do that, I ran this powershell: kubectl get secrets -n $MyNamespace | ?{$_ -match '^(\S+)\s+helm\.sh.*$'} | %{$Matches[1]} | %{kubectl delete secrets $_ -n $MyNamespace} – Prostitute

Answer 2

You can remove the helm chart forcibly by using --no-hooks flag.

Example:

helm uninstall -n <name_space> <stuck_helm_name> --no-hooks

Allogamy

This worked for me, the other solutions didn't. Thanks – Sulfur

Answer 3

Add "--debug" to the helm uninstall command it might give you a good hint (in my case it was missing permissions to delete cluster-role)

Thedathedric