How to configure kubernetes to pull container images from a registry hosted in the same cluster / inside itself

Question

I am trying to make my kubernetes cluster pull from a container registry service running inside itself. I have kube dns setup, I have a registry deployment and service running. I can resolve the service internal name via host command on the node. I have added --dns flag to docker daemon with the address of kube dns service. I have kubelet running with --cluster-dns flag set to the same address as well. Yet somehow this is what I get when I try to create a pod using this registry.

Failed to pull image "kube-registry.kube-system.svc.cluster.local/myuser/myimage": rpc error: code = Unknown desc = Error response from daemon: Get https://kube-registry.kube-system.svc.cluster.local/v1/_ping: dial tcp: lookup kube-registry.kube-system.svc.cluster.local: no such host

Somehow even with kube dns address explicitly given to both dockerd and kubelet, pulling images from the registry service fails because of name resolution. What am I missing?

Zacatecas

are you sure that the image exists in that particular registry and was pushed as kube-registry.kube-system.svc.cluster.local/myuser/myimage ? – Multiform

No. As a matter of fact, I am sure it doesn't :) But I can sort that out once I get the name resolution working. – Zacatecas

Actually... hold on, are you saying that the hostname of the registry becomes part of the image path? So I cannot push name1.domain.com/imagename and pull name2.domain.com/imagename? – Zacatecas

for internal repositories - that's often the case. in order to push, you need to tag the image with the registry host and image – Multiform

I have tested a workaround and it seems to work. I would still like to be able to refer to my registry service by name, but what I ended up doing is creating an ingress that serves localhost as hostname. Since my ingress controller runs as a daemonset, I can be very sure that every node runs it. And I use hostPort to make it map to a specific port on each node. So I can push from outside using an external name, like registry.domain.com/myuser/myapp and have kubelet/docker pull the image from the inside as localhost/myuser/myapp. This is kinda of an ugly way of doing things though. – Zacatecas

İf you'r using minikube thats solve #49639780 – Halmstad

I am not using minikube. Minikube would be easy, it has integration with host docker. – Zacatecas

Answer 1

Question: "I am trying to make my kube-cluster pull from a registry running inside itself." (Note I plan to going to edit the title of your question to clarify slightly / make it easier to search)

Short Answer: You can't*
*Nuanced Answer: Technically it is possible with hacks and a solid understanding of Kubernetes fundamentals. You'll probably want to avoid doing this, unless you have a really good reason, and fully understand my explanation of the fundamental issue and workaround, as this is an advanced use case that will require debugging to force it to work. This is complicated and nuanced enough to make step by step directions difficult, but I can give you a solid idea of the fundamental issue you ran into that makes this challenging, and high level overview guidance on how to pull off what you're trying to do anyways.

Why you can't / the fundamental issue you ran into:
In Kubernetes land 3 networks tend to exist: Internet, LAN, and Inner Cluster Network.
(Resource that goes into more depth: https://oteemo.com/kubernetes-networking-and-services-101/)
AND these 3 networks each have their own DNS / there's 3 layers of DNS.

• Internet DNS: 8.8.8.8, 1.1.1.1, 9.9.9.9 (google, cloudflare, quad9 or whatever public internet DNS the router is configured to point to.)
• LAN DNS: 192.168.1.1 (LAN DNS hosted on your router)
• CoreDNS: 10.43.0.10 (10th IP of the CIDR range of the inner cluster network)

Here's the gotcha you're running into:

• A pod can resolve DNS entries hosted at any of these 3 levels of DNS.
• The OS hosting the Kubernetes Cluster can only resolve DNS entries hosted on LAN DNS or Internet DNS. (the OS isn't scoped to have visibility into the existence of CoreDNS/Inner Cluster Network.)
• kubelet + docker/containerd/cri-o/other runtime, are responsible for pulling images from registries and these exist at the OS level in the form of systemd services and thus don't have scope to Inner Cluster DNS names. This is why what you're trying to do is failing.

Workaround options / hacks and nuances you can do to force what you're trying to do to work:

Option 1.) (I don't suggest this, has extra difficult chicken and egg issues, sharing for information purposes only)

Host an additional instance of coredns as a LAN facing instance of DNS on Kubernetes, Expose the registry and 2nd instance of coredns to the LAN via explicit NodePorts (using static service manifests so they'll come up with predictable/static NodePorts, vs random NodePorts in the range of 30000 - 32768) so they're routable from the LAN (I suggest NodePorts over LB's here as one less dependency/thing that can go wrong). Have the 2nd instance of coredns use your LAN router/LAN DNS as it's upstream DNS server. Reconfigure the OS to use the LAN facing coredns as it's DNS server.

Option 2.) More reasonable and what trow does:

• What is trow: https://thenewstack.io/trow-a-container-registry-to-run-inside-a-kubernetes-cluster/

• Proof they use the /etc/hosts method https://github.com/ContainerSolutions/trow/blob/main/QUICK-INSTALL.md

1. Pay $12 for a some-dns-name.tld

2. Use Cert Manager Kubernetes Operator or Cert Bot standalone docker container + proof you own the domain to get an https://registry.some-dns-name.tld HTTPS cert from Lets Encrypt Free. And configure your inner cluster hosted registry to use this HTTPS cert.

3. Expose the registry hosted in the cluster to the LAN using an NodePort service with an explicitly pinned convention based port number, like 32443

Why NodePort and not a LB? There's 3 reason NP is better than LB for this scenario:

• 1.) Service type LB's implementation differs between Deployment Environment and Kubernetes Distribution, while type NodePort is universal.

• 2.) If the LB changes you have to update every node's /etc/host file to point to "LB_IP registry.some-dns-name.tld" AND you have to know the LB IP, that isn't always known in advance / which means you'd have to follow some order of operations. If you use service type NodePort you can add the localhost IP entry to every node's /etc/host, so it looks like "127.0.0.1 registry.some-dns-name.tld", it's well known reusable and simplifies order of operations.

• 3.) If you ever need to change where your cluster is hosted, you can arrange it so you can make the change in 1 centralized location even in scenarios where you have no access to or control over LAN DNS. You can craft services that point to a staticly defined IP or external name (which could exist outside the cluster). and have the NodePort service point to the staticly defined service.

4. Add "127.0.0.1 registry.some-dns-name.tld" to /etc/hosts of every node in the cluster.

5. Set your yaml manifests to pull from registry.some-dns-name.tls, or configure containerd/cri-o's registry mirroring feature to map registry.some-dns-name.tld:32443 to whatever entries are being mirrored on your local registry.

6. There's 2 more solvable chicken and egg problems to deal with. 1st chicken egg problem is that Kubernetes and the registry will both likely need access to container images to even get this far.

• If you have internet access and if your internal registry is just for cacheing purposes this probably isn't a big deal.
• If you don't have internet access you'd need to .tar up the images needed by your kube distro and registry and "preload" them into spot docker/containerd/cri-o expects.
• If you don't have internet access you could alternatively have another /etc/hosts entry or LAN DNS entry for a non HA temporary docker compose based registry used for initial bootstrapping or one hosted outside the cluster.

7. 2nd chicken egg problem is the registry hosted on your cluster will need some way of being seeded with images.

• If internet access should be easy to figure out and script
• If no internet access you may need to come up with some kind of DR solution for backup and restoration of the registries backend persistent storage.
• If no internet access you could alternatively use an "emphemeral transport registry" for seeding purposes. Basically do this docker compose to spin up non-HA registry:2 image with filesystem backing, use skopeo to seed that, tar up the filesystem backing, and import it into another computer, restore the file system backing, reload non-HA registry:2 from the pre-populated filesystem backing, then use skopeo to copy from your "ephemeral transport registry" to your registry hosted in the cluster.

Actinochemistry

Update: There's an air gap kubernetes tool called zarf that makes it pretty easy to do this. It exposes a registry via nodeports and instead of node level registry configuration, uses kubernetes level registry configuration and automagically configures it using a mutating webhook (nodeport is accessible on localhost so that eliminates need for dns/hostfile edits) github.com/defenseunicorns/zarf/tree/… – Actinochemistry

Answer 2

Another solution would be to add kube-dns IP to resolv.conf:

echo "nameserver $(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')" >> /etc/resolv.conf

CoreDNS service is exposed with static IP, so there's no need to keep it updated.

I can confirm it works on Ubunutu 18.04, despite the fact that resolv.conf is generated by systemd-resolved. No additional DNS configuration was required. The services available by FQDNs only:

root@dev:~# nslookup harbor.default.svc.cluster.local
;; Got SERVFAIL reply from 127.0.0.53, trying next server
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   harbor.default.svc.cluster.local
Address: 10.109.118.191
;; Got SERVFAIL reply from 127.0.0.53, trying next server

Blub

This would only address the initial symptom and not the deeper cause/other issues you'd find after implementing this. If this was done the container runtime would be able to resolve the inner cluster dns names, but it wouldn't be able to route to correctly route to them, because the inner cluster dns names wouldn't be routable from the LAN. And kubelet leveraging docker would view routing from the perspective of the LAN. – Actinochemistry