(Kubernetes + Minikube) can't get docker image from local registry

Question

I have setup docker on my machine and also minikube which have docker inside it, so probably i have two docker instances running on different VM

I build an image and tag it then push it to local registry and it pushed successfully and i can pull it from registry too and also when i run curl to get tags list i got result, and here are what i did

1- docker build -t 127.0.0.1:5000/eliza/console:0.0.1 .
2- docker run -d -p 5000:5000 --name registry registry:2
3- docker tag a3703d02a199 127.0.0.1:5000/eliza/console:0.0.1
4- docker push 127.0.0.1:5000/eliza/console:0.0.1
5- curl -X GET http://127.0.0.1:5000/v2/eliza/console/tags/list

all above steps are working fine with no problems at all.

My problem is when i run minikube and try to access this image in local registry inside it

So when i run next commands

1- sudo minikube start --insecure-registry 127.0.0.1:5000
2- eval $(minikube docker-env)
3- minikube ssh
4- curl -X GET http://127.0.0.1:5000/v2/eliza/console/tags/list

in last step (point 4) it gave me next message

curl: (7) Failed to connect to 127.0.0.1 port 5000: Connection refused

So i can access image registry from my machine but not from minikube which make a problems of course with me when i deploy this image using Kubernetes on minikube and make deploy failed due to can't connect to http://127.0.0.1:5000

Can you help me configuring minikube to see my local registry so my problem will be solved then i can deploy image to minikube using kubernetes successfully?

UPDATE

I am using this yaml file (i named it ConsolePre.yaml) to deploy my image using kubernetes

apiVersion: v1
  kind: Service
  metadata:
    name: tripbru-console
    labels:
      app: tripbru-console
  spec:
    ports:
      - port: 9080
        targetPort: 9080
        nodePort: 30181
    selector:
      app: tripbru-console
      tier: frontend
    type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tripbru-console
  labels:
    app: tripbru-console
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: tripbru-console
        tier: frontend
    spec:
      containers:
      - image: docker.local:5000/eliza/console:0.0.1
        name: tripbru-console
        ports:
        - containerPort: 9080
          name: tripbru-console

and when i run next command to apply changes

sudo kubectl apply -f /PATH_TO_YAML_FILE/ConsolePre.yaml

the result is

NAME                                      READY     STATUS         RESTARTS   AGE
po/tripbru-console-1655054400-x3g87       0/1       ErrImagePull   0          1m

and when i run describe command

sudo kubectl describe pod tripbru-console-1655054400-x3g87

i found next message in description result

Error response from daemon: {"message":"Get https://docker.local:5000/v1/_ping: dial tcp: lookup docker.local on 10.0.2.3:53: read udp 10.0.2.15:57792-\u003e10.0.2.3:53: i/o timeout"}

and i configured docker.local xxx.xxx.xx.4 in minikube /etc/hosts so i don't know from where 10.0.2.3:53 and 10.0.2.15:57792 come from.

So how can i solve this issue too.

Thanks :)

Cryptography

Are you able to ping docker.local from insider the minikube VM? – Subliminal

@TarunLalwani yes and i got result of ping like that [64 bytes from 192.168.1.4: seq=1 ttl=63 time=0.715 ms] – Cryptography

Create a file /etc/docker/daemon.json with contents {"insecure-registries": ["docker.local:5000", "xxx.xxx.xx.4:5000"]}. Use the IP also. After creating the file restart docker and test your system again. This all will be done inside Minukube VM – Subliminal

@TarunLalwani i created /etc/docker/daemon.json inside minikube and added {"insecure-registries": ["docker.local:5000", "192.168.1.4:5000"]} to it but when i try to restart docker it failed and i got this >> level=fatal msg="unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: insecure-registries: (from flag: [10.0.0.0/24], from file: [docker.local:5000 192.168.1.4:5000])\n" – Cryptography

Please check the edits in my answer – Subliminal

Answer 1

The issue is your notion using 127.0.0.1 anywhere you want. This is wrong.

So if your machine IP is 192.168.0.101. Then below works

1- docker build -t 127.0.0.1:5000/eliza/console:0.0.1 .
2- docker run -d -p 5000:5000 --name registry registry:2
3- docker tag a3703d02a199 127.0.0.1:5000/eliza/console:0.0.1
4- docker push 127.0.0.1:5000/eliza/console:0.0.1
5- curl -X GET http://127.0.0.1:5000/v2/eliza/console/tags/list

Because docker run maps the registry to 127.0.0.1:5000 and 192.168.0.101:5000. Now on your machine only this 127.0.0.1 will work. Now when you use

3- minikube ssh

You get inside the minikube machine and that doesn't have a registry running on 127.0.0.1:5000. So the error. The registry is no reachable inside this machine using the machine IP.

The way I usually solve this is issue is by using host name both locally and inside the other VMs.

So on your machine create a entry in /etc/hosts

docker.local 127.0.0.1

And change your commands to

1- docker build -t docker.local:5000/eliza/console:0.0.1 .
2- docker run -d -p 5000:5000 --name registry registry:2
3- docker tag a3703d02a199 docker.local:5000/eliza/console:0.0.1
4- docker push docker.local:5000/eliza/console:0.0.1
5- curl -X GET http://docker.local:5000/v2/eliza/console/tags/list

And then when you use minikube ssh, make a entry for docker.local in /etc/hosts

docker.local 192.168.0.101

Then curl -X GET http://docker.local:5000/v2/eliza/console/tags/list

Edit-1

For the TLS issue you need to Stop the docker service inside minikube

systemctl stop docker

Then edit /etc/systemd/system/docker.service.d/10-machine.conf and change

ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.0.0.0/24

to

ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=virtualbox --insecure-registry 10.0.0.0/24 --insecure-registry docker.local:5000 --insecure-registry 192.168.1.4:5000

Then reload daemon and start the docker service

systemctl daemon-reload
systemctl start docker

After that try to pull

docker pull docker.local:5000/eliza/console:0.0.1

And the command should work

Subliminal

Thanks for your answer and actually it helped me alot :) and now i can access url from minikube ssh, but my main problem actually in getting image from kubernetes, please check my question update and i hope you can help me too in that – Cryptography

i can't get this file /etc/systemd/system/docker.service.d/10-machine.conf inside minikube v0.22.1, are they allocated it in new path? i can only see same content of old file (10-machine.conf) inside this file /usr/lib/systemd/system/docker.service but when i add --insecure-registry to it and start docker again, i can't pull images from my local registry – Cryptography

You check where the dropin file is located. systemctl status docker – Subliminal

I had to modify docker.service on the minikube machine – Kafka

Answer 2

How to access Processes running on hostmachine from with in a Docker container?

It is a popular question in the docker-land. See here. https://mcmap.net/q/40392/-from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-machine There are other ways too, for example, For Docker on mac, docker.for.mac.localhost DNS name will resolve to the hostmachine

From https://docs.docker.com/docker-for-mac/networking/#i-cannot-ping-my-containers

The Mac has a changing IP address (or none if you have no network access). From 17.06 onwards our recommendation is to connect to the special Mac-only DNS name docker.for.mac.localhost which will resolve to the internal IP address used by the host.

Assuming that primary purpose of this minikube is for local testing, there is an easier way deploy your docker container (This doesnt even need a local docker registry)

Method 2: Point your docker CLI to Docker daemon running within your minikube and then execute docker build command there.

First thing to understand here is, when you install docker in your machine, it has 2 parts, 1) a docker cli with which you can interact with docker daemon 2) A docker daemon. In this method we will point our local docker cli to minikube's docker daemon and execute docker build.

https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/getting-started-guides/minikube.md#reusing-the-docker-daemon

quoting relevant parts here

When using a single VM of Kubernetes, it's really handy to reuse the minikube's built-in Docker daemon; as this means you don't have to build a docker registry on your host machine and push the image into it - you can just build inside the same docker daemon as minikube which speeds up local experiments. Just make sure you tag your Docker image with something other than 'latest' and use that tag while you pull the image. Otherwise, if you do not specify version of your image, it will be assumed as :latest, with pull image policy of Always correspondingly, which may eventually result in ErrImagePull as you may not have any versions of your Docker image out there in the default docker registry (usually DockerHub) yet.

To be able to work with the docker daemon on your mac/linux host use the docker-env command in your shell:

eval $(minikube docker-env)

You should now be able to use docker on the command line on your host mac/linux machine talking to the docker daemon inside the minikube VM:

do a docker container list command : docker ps. It should display even the containers related to kubernetes system (because now your cli is pointed to a docker daemon where your minikube is running).

Now build your docker image. Then it will be available in the minikube for you.

Edwyna

"make sure you tag your Docker image with something other than 'latest' and use that tag while you pull the image": this did it for me. – Roundtheclock

Thanks @Andre.. That's actually my PR which updated the official documentation. Happy to see that it helped you! – Edwyna

Answer 3

Minikube runs inside a docker container, so you should see it as a separate machine. Now, inside this machine Kubernetes runs, NOTE it is important to understand that we have the Minikube environment and the Kubernetes environment. This is important to understand because it is not the same to connect to the local registry from Minikube than from Kubernetes (which resides on Minikube). Here the difference: enter image description here

When you create a Job / Deployment / Statefulset, the creation is done by minikube, so it does not know if there is any service that connects to our local registry in docker. Curiously, our "registry" service does work within pods, that is, once our Job / Deployment / Statefulset has been created, then you can access our "registry" service without problems. So what is the solution to all this? Minikube can easily connect to our local registry, through 192.168.49.1:5000. If you want your Jobs / Deployments / Statefulsets to be created with images from a local registry, then just add 192.168.49.1:5000 to your image and voila. enter image description here On the other hand, if you want to be able to access your local registry from within pods, you will need a service and an endpoint.

Considerations: Remember that it is very important that you allow minikube to access your local registry by:

minikube start --insecure-registries 192.168.49.1:5000

It is rare that minikube uses another ip other than 192.168.49.1, just in case it is better to check with:

minikube ssh 'grep host.minikube.internal /etc/hosts | cut -f1 '

This is all assuming you have a registry created in docker with port 5000 exposed.

Except

I liked the previous solution using eval $(minikube docker-env), but connecting from minikube to my outer docker registry using minikube start --insecure-registries 192.168.49.1:5000 as show here is nicer when also working with plain docker at times. Both works (also on WSL2). Thanks! – Hughes

Answer 4

You can issue this command to point your docker CLI to minikube: eval $(minikube docker-env) And then you can build your images there or export them from wherever you have them and import.

Litigable