What is the difference between Kaniko and BuildKit/Buildx?

Question

From what I understand:

• They are both tools to build container images
• The build itself runs in a container
• The build can happen on a remote node, for example in a Kubernetes cluster (Kaniko, BuildKit)
• They both offer advanced features such as layer caching

The differences I can gather:

• Security model (Kaniko)
• BuildKit leverages more recent developments such as cache manifest and manifest lists
• BuildKit supports multiple architectures

What I'm not clear is the extent of the overlap between the 2 set of tools and when one should be used instead of the other. For example, both tools seem to cover well the use case of self hosting a remote image build farm on a Kubernetes cluster.

Peugia

Answer 1

Overlapping features notwithstanding, the primary differences are these:

	BuildKit	Kaniko
build with no root or daemon²	✔	✔
build multi-architecture³	✔
remote layer caching⁴	✔	✔
local layer caching⁵	✔

² Both Kaniko and BuildKit can run daemonless and rootless, though Kaniko is, practically speaking and in my humble opinion, easier to build a container from within a non-root container. Kaniko "builds as a root user within a container in an unprivileged environment", but does not require root or a daemon. BuildKit, when exposed via buildx, requires a privileged docker daemon, but BuildKit requires no daemon or root privileges in its standalone form (with some tooling like RootlessKit).

³ Kaniko does not support multi-architecture builds at the time of writing this. https://docs.docker.com/desktop/multi-arch/#build-multi-arch-images-with-buildx

⁴ BuildKit and Kaniko support registry-based caching. BuildKit, however, requires the registry have support for cache manifest lists.

⁵ BuildKit supports multiple --cache-to options, including local filesystem. https://docs.docker.com/engine/reference/commandline/buildx_build/#cache-to

Typically the restraints / features of your build environment or platform would dictate which tool is most appropriate, and if you have both as an option, speed may help you decide (though this should be benchmarked thoroughly).

Morrow

buildx is a convenience wrapper for docker users to access buildkit, and buildkit itself runs in a container, no docker engine needed for that lower level. – Halfbaked

@Halfbaked thank you. I've updated the answer to correct this and make the point you brought up more clear. – Morrow

Another big gotcha is neither Buildah or Kaniko support the latest Dockerfile syntax. This means you will be left with devs scratching their heads why images build with buildkit but not kaniko/buildah. For example kaniko ignores the --chmod flag in ADD instructions. – Prevocalic