helm rollback fails to identify the failed deployments when re-triggered

Question

I have a scenario like below, Have two releases - Release-A and Release-B.

Currently, I am on Release-A and need an upgrade of all the microservices to Release-B. I tried performing the helm upgrade of microservice - "mymicroservice" with the below command to deliver Release-B.

helm --kubeconfig /home/config upgrade --namespace testing --install --wait mymicroservice mymicroservice-release-b.tgz

Because of some issue, the deployment object got failed to install and went into an error state.

Observing this, I perform the below rollback command.

helm --kubeconfig /home/config --namespace testing rollback mymicroservice

Due to some issue(may be an intermittent system failure or user behavior), the Release-A's deployment object also went into failed/Crashloopbackoff state.Although this will result in helm rollback success, the deployment object is still not entered the running state.

Once I made the necessary corrections, I will retry the rollback. As the deployment spec is already updated with helm, it never attempts to re-install the deployment objects even if it is in the failed state.

Is there any option with Helm to handle the above scenarios ?.

Tried with --force flag, but there are other errors related to Service object replace in the microservice when used the --force flag approach.

Rollback "mymicroservice -monitoring" failed: failed to replace object: Service "mymicroservice-monitoring" is invalid: spec.clusterIP: Invalid value: "": field is immutable

Spadework

There is a suggestion to try --force flag with helm latest 3.5.4, but facing a similar issue. Is the --force flag triggers with helm delete --purge and re-installs the service ?. – Peggie

Answer 1

Maybe this can help u out:

• Always use the helm upgrade --install command. I've seen you're using so you're doing well. This installs the charts if they're not present and upgrades them if they're present.
• Use --atomic flag to rollback changes in the event of a failed operation during helm upgrade.
• And the flag --cleanup-on-fail: It allows that Helm deletes newly created resources during a rollback in case the rollback fails.

From doc:

--atomic: if set, upgrade process rolls back changes made in case of failed upgrade. The --wait flag will be set automatically if --atomic is used

--cleanup-on-fail allow deletion of new resources created in this upgrade when upgrade fails

There are cases where an upgrade creates a resource that was not present in the last release. Setting this flag allows Helm to remove those new resources if the release fails. The default is to not remove them (Helm tends to avoid destruction-as-default, and give users explicit control over this)

https://helm.sh/docs/helm/helm_upgrade/

Knox

Answer 2

IIRC, helm rollback rolls back to the previous version, whether it is good or not, so if your previous attempts resulted in a failure and you try to rollback, you will rollback to a broken version

Greenbrier