helmfile sync vs helmfile apply

Question

sync      sync all resources from state file (repos, releases and chart deps)
apply     apply all resources from state file only when there are changes

sync

The helmfile sync sub-command sync your cluster state as described in your helmfile ... Under 
the covers, Helmfile executes helm upgrade --install for each release declared in the 
manifest, by optionally decrypting secrets to be consumed as helm chart values. It also 
updates specified chart repositories and updates the dependencies of any referenced local 
charts.

For Helm 2.9+ you can use a username and password to authenticate to a remote repository.

apply

The helmfile apply sub-command begins by executing diff. If diff finds that there is any changes
sync is executed. Adding --interactive instructs Helmfile to request your confirmation before sync.
An expected use-case of apply is to schedule it to run periodically, so that you can auto-fix skews
between the desired and the current state of your apps running on Kubernetes clusters.

I went through the Helmfile repo Readme to figure out the difference between helmfile sync and helmfile apply. It seems that unlike the apply command, the sync command doesn't do a diff and helm upgrades the hell out of all releases 😃. But from the word sync, you'd expect the command to apply those releases that have been changed. There is also mention of the potential application of helmfile apply to periodically syncing of releases. Why not use helmfile sync for this purpose? Overall, the difference didn't become crystal clear, and I though there could probably be more to it. So, I'm asking.

Tish

Answer 1

Consider the use case where you have a Jenkins job that gets triggered every 5 minutes and in that job you want to upgrade your helm chart, but only if there are changes.

If you use helmfile sync which calls helm upgrade --install every five minutes, you will end up incrementing Chart revision every five minutes.

$ helm upgrade --install httpd bitnami/apache > /dev/null
$ helm list
NAME    REVISION        UPDATED                         STATUS          CHART           APP VERSION     NAMESPACE
httpd   1               Thu Feb 13 11:27:14 2020        DEPLOYED        apache-7.3.5    2.4.41          default
$ helm upgrade --install httpd bitnami/apache > /dev/null
$ helm list
NAME    REVISION        UPDATED                         STATUS          CHART           APP VERSION     NAMESPACE
httpd   2               Thu Feb 13 11:28:39 2020        DEPLOYED        apache-7.3.5    2.4.41          default

So, each helmfile sync will result new revision. Now if you were to run helmfile apply, which will first check for diffs and only then (if found) will call helmfile sync which will in turn call helm upgrade --install this will not happen.

Markettamarkey

so we should just use helmfile apply instead of sync in general ? – Gaivn

@Gaivn Yes, I have rarely used sync, while apply I use daily. – Markettamarkey

There is one important implication of sync command. If someone deletes some deployment or configmap from the chart manually, apply will not see any changes and that resource will still be deleted, while sync will recreate it restoring original chart configuration. – Shaunda

@lumaks, is that intended behaviour? Why would deployments and configmaps be missed by diff? Surely if manual changes are made to the environment, a periodic "apply" should bring them back in line with the source code? – Pythagorean

Yeah, it's intended because apply will only execute helm upgrade --install if helm diff returns some changes. And with manual operations on the cluster, helm diff won't return any changes, so running sync is the best way to keep the state in sync with the code. – Shaunda

Answer 2

Everything in the other answers is correct. However, there is one additional important thing to understand with sync vs apply:

• sync will call helm upgrade on all releases. Thus, a new helm secret will be created for each release, and releases' revisions will all be incremented by one. Helm 3 uses three-way strategic merge algorithm to compute patches, meaning it will revert manual changes made to resources in the live cluster handled by Helm;
• apply will first use the helm-diff plugin to determine which releases changed, and only run helm upgrade on the changed ones. However, helm-diff only compares the previous Helm state to the new one, it doesn't take the state of the live cluster into account. This means if you modified something manually, helmfile apply (or more precisely the helm-diff plugin) may not detect it, and leave it as-is.

Thus, if you always modify the live cluster through helm, helmfile apply is the way to go. However, if you can have manual changes and want to ensure the live state is coherent with what is defined in Helm/helmfile, then helmfile sync is necessary.

---

If you want more details, checkout my blog post helmfile: difference between sync and apply (Helm 3)

Ethnic

There's an update on the article about three way merge diffs using HELM_DIFF_THREE_WAY_MERGE env var. Using HELM_DIFF_THREE_WAY_MERGE=true helmfile apply helmfile can now detect and change manual changes as well. If you stick with this best practice, apply is always the way to go. (text copied from source) – Hooked

Answer 3

In simple words, this is how both of them works:

• helmfile sync calls helm upgrade --install

• helmfile apply calls helm upgrade --install if and only if helm diff returns some changes.

So, generally, helmfile apply would be faster and I suggest using it most of the time.

But, put in your consideration that if someone has deleted any deployments or configmaps manually from the chart, helm diff won't see any changes hence,helmfile apply won't work and these resources will still be deleted, while helmfile sync will recreate it restoring original chart configuration.

Eudo

Answer 4

We have detected one significant issue that also has implications.

Sometimes a sync or apply operation can fail due to:

• Timeout with wait: true. E.g. k8s cluster needs to add more nodes and operation takes longer than expected (but eventually everything is deployed).
• Temporary error on a postsync hook.

In these cases a simple retry on the deployment job of the pipeline would solve the issue but succesive helmfile apply will skip both helm upgrade and hook execution, even if release is in status failed.

So my conclusions are:

• apply is usually faster but can lead to situations where manual (outside CI/CD logic) are required.
• sync is more robust (CI/CD friendly) but usually slower.

Juicy