How to clean-up old unused Kubernetes images/tags?

Question

To simplify deployment and short term roll-back, it's useful to use a new Docker image tag for each new version to deploy on Kubernetes. Without clean-up this means that old images:tags are kept forever.

How can I list all image:tag that are used by a Kubernetes container so that I can find all old image:tag that are old and not used to delete them automatically from the Docker Registry?

My goal is ideally for Google Container Engine (GKE) to delete unused images a Google Container Registry.

Redford

gist.github.com/ahmetb/7ce6d741bd5baa194a3fac6b1fec8bb7 Script to clean up Google Container Registry images pushed before a particular date – Biamonte

Answer 1

As an alternative approach, you might consider just letting Kubernetes handle reclamation of old images for you.

Presently, the ImageManager handles reclamation of candidate images. See: Garbage Collection

Garbage collection is a helpful function of kubelet that will clean up unreferenced images and unused containers. kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.

Configuration is controlled via these two kublet cli parameters:

  --image-gc-high-threshold=90: The percent of disk usage after which image garbage collection is always run. Default: 90%
  --image-gc-low-threshold=80: The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Default: 80%

The high/low thresholds could be tuned to force collection at an interval that works for you.

Delubrum

Isn't that only for local images on the cluster Nodes and not for images in the Docker Registry? – Redford

Sorry I missed to specify it in my question properly. Updated the question. – Redford

Answer 2

With recent versions of kubelet use the below options as --image-gc-high-threshold and --image-gc-low-threshold are being deprecated:

--eviction-hard
--eviction-soft

More details avaialble here:

• https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/
• https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

Bronchitis

Answer 3

To get a list of all images used by a Kubernetes cluster, one can run the shell script:

for image in $(kubectl get pods --all-namespaces --output=jsonpath='{..image}')
do
    echo $image
done

It seems however that there is no way to simply currently delete images from a Google Container Registry (see How to remove a pushed image in Google Container Registry)

Redford

If you need to know which images are still referenced by your cluster, and assuming you use Deployments rather than create Pods directly, you could look at ReplicaSets instead of Pods by replacing the "pods" with "rs" in your command. – Compound

The above in a sorted list with occurrence-count (kubernetes.io/docs/tasks/access-application-cluster/…): kubectl get pods --all-namespaces -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq -c – Fachini

Answer 4

You could use docker-cleanup containers running in a DaemonSet. That would cleanup any unused images on each node in your cluster.

Magnitude

Answer 5

I am not sure if there is a documented approach to do that kind of maintenance. However Openshift Origin does attempt to tackle by pruning docker images and interacting with registry to remove older blobs

We have implemented it in context to origin. The source code for that it on github

Fellers

Answer 6

A new feature for modern kubernetes (v1.29+) for automatic cleanup by the Kubelet (i.e. garbage collection) is the two Kubelet configuration options:

imageMaximumGCAge: <max_unused_time>
imageMinimumGCAge: <min_unused_time>

which allow you to clean up container image when they are considered old enough after being unused for that max threshold (as opposed to waiting for max disk usage in @Ryan Cox's answer).

Sources:

• https://kubernetes.io/docs/concepts/architecture/garbage-collection/#containers-images
• https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

Didymous