How to use local docker images in kubernetes deployments (NOT minikube)

Question

I have a VM with kubernetes installed using kubeadm (NOT minikube). The VM acts a the single node of the cluster, with taints removed to allow it to act as both Master and Worker node (as shown in the kubernetes documentation). I have saved, transfered and loaded a my app:test image into it. I can easily run a container with it using docker run. It shows up when I run sudo docker images.

When I create a deployment/pod that uses this image and specify Image-PullPolicy: IfNotPresent or Never, I still have the ImagePullBackoff error. The describe command shows me it tries to pull the image from dockerhub...

Note that when I try to use a local image that was pulled as the result of creating another pod, the ImagePullPolicies seem to work, no problem. Although the image doesn't appear when i run sudo docker images --all.

How can I use a local image for pods in kubernetes? Is there a way to do it without using a private repository?

Synchronism

which kubernetes version are you using? – Senile

I am using version v1.22 – Synchronism

Answer 1

One can do so with a combination of crictl and ctr, if using containerd.

TLDR: these steps, which are also described in the crictl github documentation:

1- Once you get the image on the node (in my case, a VM), make sure it is in an archive (.tar). You can do that with the docker save or ctr image export commands.

2- Use sudo ctr -n=k8s.io images import myimage.tar while in the same directory as thearchived image to add it to containerd in the namespace that kubernetes uses to track it's images. It should now appear when you run sudo crictl images.

---

As suggested, I tried listing images with crictl and my app:test did not appear. However, trying to import my local image through crictl didn't seem to work either. I used crictl pull app:test and it showed the following error message:

FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/app:test": failed to resolve reference "docker.io/library/app:test": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed.

However, when following these steps, my image is finally recognized as an existing local image in kubernetes. They are actually the same as suggested in the crictl github documentation

How does one explain this? How do images get "registered" in the kubernetes cluster? Why couldn't crictl import the image? I might post another issue to ask that...

Synchronism

As external links tend to disappear, it would be nice to summarize what exactly "these steps" are in this answer. – Metrist

Noted, editing now. – Synchronism

Answer 2

In my case, I have setup k8s cluster with kubeadm. so, if you want to use local image to run pod do below things.

If you have docker image, create tar file of image by using below command.

docker save <image-name> -o <filename.tar>

If you have containerd image, create tar file of image by using below command.

ctr image export <output-filename> <image-name>

After transferring the standalone archives to the other systems (using whatever means you prefer; I used scp), then load (or import) the images into containerd with this command:

ctr -n=k8s.io images import <filename-from-previous-step>

Now you can see image.

sudo crictl images

Hermaherman

Answer 3

image doesn't appear when i run sudo docker images --all

Based on your comment, you are using K8s v1.22, which means it is likely your cluster is using containerd container runtime instead of docker (you can check with kubectl get nodes -o wide, and see the last column).

Try listing your images with crictl images and pulling with crictl pull <image_name> to preload the images on the node.

Senile

I will try. Doesn't docker use containerd as the runtime under the hood? That's why I used containerd for my node; so that i could rely on docker for information on my images. Was I wrong? – Synchronism

This is correct, docker does use containerd itself, it is just k8s doesn't use docker as a middle man anymore. Check and see if you can get further with crictl. You can see an example of listing images by docker and containerd on one of my k8s nodes here. – Senile

Thanks. Running crictl images, my image indeed doesn't pop up. However, when I try to pull it I get this error message: FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/app:test": failed to resolve reference "docker.io/library/app:test": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed – Synchronism

Using crictl did not work, but following these steps worked correctly... Any idea why crictl may not have been able to pull? – Synchronism

This seems like a private repo? – Senile

If this post is any indication, it is actually a public one; the default one that is set when no other is specified... It explains why it asks for an authorizaiton. It seems crictl pull cannot actually take images from the local machine directly... I haven't found any official confirmation of this, though. – Synchronism

Answer 4

Your cluster is bottled inside of your VM, so what you call local will always be remote for that cluster in that VM. And the reason that kubernetes is trying to pull those images, is because it can't find them in the VM.

Dockerhub is the default place to download containers from, but you can set kubernetes to pull from aws (ECR) from azure (ACR), from github packages (GCR) and from your own private server.

You've got about 100 ways to solve this, none of them are easy or will just work.

1 - easiest, push your images to Dockerhub and let your cluster pull from it.

2 - setup a local private container registry and set your kubernetes VM to pull from it (see this)

3 - setup a private container registry in your kubernetes cluster and setup scripts in your local env to push to it (see this)

Depression

What exactly do you mean by "bottled"? I did put the image on the VM... That is the source of my confusion. And other images pulled by Kubernetes can be reused even though they don't show up on the VM. How So? – Synchronism

bottled in the sense of encapsuled, your VM doesn't have access to your machine and your machine doesn't have access to your VM, unless you access it using ssh or mount folders, but that's all networking. In that sense of bottled, your VM can't see your machines's images and vice verse. – Depression

when you mean 'show up on the VM' do you mean by running 'docker image ls' ? if so, that's probably because kubernetes is using a different container driver. Docker is only one way of interacting with containers, it's the most famous and real hardcore container linux people hate docker because docker is a company and it has its own agenda. I think kubernetes is using CRI-O as the default container runtime now, try running crictl images to see the images that kubernetes pulls – Depression

Ok, well in that case I have to say: this is not the issue. As stated in my question, I did transfer the image onto my VM and loaded it there. Running docker images shows it on the VM. Yes, that is what I mean. When I set up kubernetes, I followed the instructions to use containerd as a runtime. Docker uses containerd as well, so docker images should be relevant, no? – Synchronism

got it, so yeah, the images that kubernetes uses should all be showing up on your VM, silly question, are you using the full label with image version or just the image name? could you update your question with the deployment or pod spec? let's get to the bottom of this – Depression

I progressed quite a bit following the crictl tip. Loading with ctr directly yields results. I guess CRI-O works somewhat similarly, seeing as it was built for kubernetes. – Synchronism

Answer 5

Here is another solution that worked for me.

The easiest solutions was in the end to install podman and load the image. I'm running on manjaro, using CRI-O as my CRI and kubeadm as control-plane.

sudo pacman -Sy podman 

save your image from docker to a .tar file and load the image using sudo podman. (Note: running podman with sudo is necessary)

sudo podman load -i your-local-image.tar

It instantly showed up in the CRI-image registery when running

sudo crictl images

Example to define your kubernetes job:

#local-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: local-job
spec:
  template:
    spec:
      containers:
      - name: local-img-pod
        image: localhost/<Your-image-tag> 
        command: ["./your-app-binary","parameter1"]
        imagePullPolicy: Never
      restartPolicy: Never
  backoffLimit: 4

and start the job with

kubectl apply -f local-job.yaml

Unknown