"crictl rmi" removing all the images which are in use

Question

I'm trying to remove all unused images with specific name format from Kubernetes cluster like below.

crictl images | grep -E -- 'foo|bar' | awk '{print \$3}' | xargs -n 1 crictl rmi

But this one also deletes all the images with naming "foo" or "bar" even it's in use by container. Tried using "crictl rmi -q" but that deletes multiple other images which are not in the filter above.

Hernando

Why do you want to do that? – Oration

@Oration - As we continuously update the containers with new images, the old one still remains in the cluster as cached. We don't want to keep them and want to remove all unused one. But above command also delete the used images. If we delete those also then when container restart for any reason then it needs to pull the image again and again, which takes time instead of using cached image. – Hernando

Kubernetes (kubelet) automatically cleans up unused images based on disk usage: kubernetes.io/docs/concepts/architecture/garbage-collection/… and you can even tune the thresholds – Oration

I can't use that option as my main concern is not only disk space. I also need to keep up with security patches of base linux OS. So I can't let old images cached in the cluster. – Hernando

Why is an old image, that is not used until garbage collected, a concern for you? Do you want to remove containers or images? – Oration

As a sample we have foo:100 and now we are updating container with foo:101. But the k8s keeps the foo:100 in the cache images list. So, after updating container to foo:101 we don't want to keep foo:100 as it might contains old security patches which will can create issue for us even it's not in use. But if I run above command it deletes foo:100 as well foo:101. Which is not a correct behavior. Previously when we were using "docker rmi" it was only deleting images which are not in use by containers. – Hernando

How can an unused image lead to security problems? – Oration

As the images use linux as base os and we continuously gets security patches for linux. So if I keep the old images in k8s cluster our monitors running in the environments flag those images as vulnerable – Hernando

Answer 1

Probably you want to run

crictl rmi --prune

You need a rather current crictl for that. From the help:

$ crictl rmi --help
NAME:
   crictl rmi - Remove one or more images

USAGE:
   crictl rmi [command options] IMAGE-ID [IMAGE-ID...]

OPTIONS:
   --all, -a    Remove all images (default: false)
   --prune, -q  Remove all unused images (default: false)
   --help, -h   show help (default: false)

Oration

I tried --prune but that also deletes all the images in my case. Also if I try like below, it's not even using filtered images and delete all in the k8s cluster. crictl images | grep -E -- 'foo|bar' | awk '{print \$3}' | xargs -n 1 crictl --prune – Hernando

It will only remove unused images with --prune, and since your requirement is to not have laying around any old images with potential security problems it should solve your use case. Does it not? – Oration

I ran "crictl images" first to get the list of images. Then I ran "crictl rmi --purne". ANd then again when I run "crictl images" it saws empty list. As well I verified from console output that "--purne" is deleting the image which is already in use by container. – Hernando

You should not run "crictl images" first, as this will list all images, whether in use or not. Instead you should run "crictl rmi --prune" on its own. – Crocoite

Answer 2

If your version of crictl does not have the --prune option, you can use the following command:

yum -y install jq
comm -23 <(crictl images  -q | sort) <(crictl ps -q | xargs -n 1 crictl inspect -o json | jq -r '.info.config.image.image' | sort) | xargs crictl rmi

comm usage:

# 1 < 2
comm -13 <(cat 1) <(cat 2)
 
# 1 > 2
comm -23 <(cat 1) <(cat 2)

Prepared