Currently, running a private Docker registry (Artifactory) on an internal network that uses a self signed certificate for authentication.
When Kubernetes starts up a new node, it is unable to auth with the private Docker registry because this new node does not have the self signed certificate.
Any help would be much appreciated. Thanks!
The simplest solution I found after an extensive search is suggested in this guide by CoreOS : https://github.com/coreos/tectonic-docs/blob/master/Documentation/admin/add-registry-cert.md
It consists to create a secret that contains your certificate and a DaemonSet to populate it to /etc/docker/certs.d/my-private-insecure-registry.com/ca.crt on all the nodes of your cluster.
I think this answers your question because, when adding a new node, the DaemonSet is automatically executed on it.
I give the detailed solution below but all the credits goes to Kyle Brown (kbrwn) for his very cool guide (cf. link above).
Lets suppose that your certificate is a file named ca.crt in your working directory. Create a secret from this file content :
kubectl create secret generic registry-ca --namespace kube-system --from-file=registry-ca=./ca.crt
Then, use the following DaemonSet that mounts the certificate as the file /home/core/registry-ca and copy it to the desired location : /etc/docker/certs.d/reg.example.com/ca.crt.
Simply replace my-private-insecure-registry.com with the hostname of your container registry.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: registry-ca
namespace: kube-system
labels:
k8s-app: registry-ca
spec:
selector:
matchLabels:
name: registry-ca
template:
metadata:
labels:
name: registry-ca
spec:
containers:
- name: registry-ca
image: busybox
command: [ 'sh' ]
args: [ '-c', 'cp /home/core/registry-ca /etc/docker/certs.d/my-private-insecure-registry.com/ca.crt && exec tail -f /dev/null' ]
volumeMounts:
- name: etc-docker
mountPath: /etc/docker/certs.d/my-private-insecure-registry.com
- name: ca-cert
mountPath: /home/core
terminationGracePeriodSeconds: 30
volumes:
- name: etc-docker
hostPath:
path: /etc/docker/certs.d/my-private-insecure-registry.com
- name: ca-cert
secret:
secretName: registry-ca
Save the file as registry-ca-ds.yaml and then create the DaemonSet :
kubectl create -f registry-ca-ds.yaml
You can now check that your application correctly pulls from your private self-signed registry.
As mentioned, the certificate will be added to new nodes' docker in an automatic fashion by the registry-ca DaemonSet. If you want to avoid this, simply delete the DaemonSet :
kubectl delete ds registry-ca --namespace kube-system
I think this is more secure than setting the insecure-registries flag of the docker daemon. Also, it is resilient to new nodes.
The Kubernetes project is removing built-in support for the Docker runtime in Kubernetes version 1.24 and later (source). Therefore, these solutions doesn't work for Kubernetes 1.24+ clusters that should use Containerd instead of Docker. As explained in Moly's answer, Containerd doesn't support yet adding certificates without restarting so a different approach using a privileged container is recommended for those clusters.
Please, see Moly's answer for detailed explanation and code.
According to these Kubernetes answers (here and here) to related github issues, a Kubernetes volume path cannot contain a colon. Therefore, this solution is not valid for registries that communicate securely with a self-signed certificate on specific ports (for example 5000).
In such case, please see Gary Plattenburg's answer to create the directory in the shell command instead of using Kubernetes to handle it during the mount.
Recent versions of kubernetes now use containerd instead of docker to pull images, so the other answers will no longer work. You can check which your nodes are using by running kubectl get nodes -o wide and looking under "CONTAINER-RUNTIME".
Currently the only way I can find to get this working with containerd is to add the certificate to the root store of the host and then restart the containerd service. Doing this via a DaemonSet requires using a privileged container and nsenter so that we can run a shell on the host. This config worked for me for an Ubuntu host (create the secret with your certificate first):
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: registry-ca
namespace: kube-system
labels:
k8s-app: registry-ca
spec:
selector:
matchLabels:
name: registry-ca
template:
metadata:
labels:
name: registry-ca
spec:
hostPID: true
hostNetwork: true
initContainers:
- name: registry-ca
image: busybox
securityContext:
privileged: true
command: [ 'sh' ]
args:
- -c
- |
cp /home/core/registry-ca /usr/local/share/ca-certificates/registry-ca.crt
nsenter --mount=/proc/1/ns/mnt -- sh -c "update-ca-certificates && systemctl restart containerd"
volumeMounts:
- name: usr-local-share-certs
mountPath: /usr/local/share/ca-certificates
- name: ca-cert
mountPath: /home/core
terminationGracePeriodSeconds: 30
volumes:
- name: usr-local-share-certs
hostPath:
path: /usr/local/share/ca-certificates
- name: ca-cert
secret:
secretName: registry-ca
containers:
- name: wait
image: k8s.gcr.io/pause:3.1
There is currently an open merge request for containerd that should allow adding certificates without restarting, so hopefully we can use something similar to Bichon's simpler answer which doesn't need the host access workarounds soon.
You basically have to tell the Docker daemon to trust your self-signed certificate by telling it to trust the Certificate Authority (CA) that you used to sign the certificate. You can find more information here on the section that says "Use self-signed certificates".
In particular for example for Linux:
Linux: Copy the
domain.crtfile to/etc/docker/certs.d/myregistrydomain.com:5000/ca.crton every Docker host. You do not need to restart Docker.
This all different from authenticating by specifying ImagePullSecrets on your pods or docker login credentials in your docker config files.
The accepted answer from Bichon fro 1.16+ has in the comments that it does not work for URLs on a port due to not being able to mount the path with a colon (:) in it. This can be fixed by mounting the parent directory and changing the command arguments to make it.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: registry-ca
namespace: kube-system
labels:
k8s-app: registry-ca
spec:
selector:
matchLabels:
name: registry-ca
template:
metadata:
labels:
name: registry-ca
spec:
containers:
- name: registry-ca
image: busybox
command: [ 'sh' ]
args: [ '-c', 'mkdir /etc/docker/certs.d/my-private-insecure-registry.com:5000 && cp /home/core/registry-ca /etc/docker/certs.d/my-private-insecure-registry.com:5000/ca.crt && exec tail -f /dev/null' ]
volumeMounts:
- name: etc-docker
mountPath: /etc/docker/certs.d
- name: ca-cert
mountPath: /home/core
terminationGracePeriodSeconds: 30
volumes:
- name: etc-docker
hostPath:
path: /etc/docker/certs.d
- name: ca-cert
secret:
secretName: registry-ca
You can access the keys for private docker registries in $HOME/.dockercfg or $HOME/.docker/config.json . If you add it to one of these search paths kubelet should use it as a credential when pulling the images.
https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry
The "Configuring Nodes to Authenticate to a Private Registry" section gives you a step by step on how to do it.
Kubernetes is likely using the docker daemon on the Kubernetes cluster nodes. For them to trust your local registry, you can the trusted registry hostnname to the file /etc/docker/daemon.json as follows:
{ "insecure-registries":["some.local.registry"] }
where some.local.registry is the hostname of the registry.
You need to restart the docker process(es) to make this effective. I did this for a domain that is not public and has no valid TLD, so I could not use cert-manager with letsencrypt.
You need to do the same on every machine that uses docker to connect to that registry.
docker login command on all the nodes. –
Creigh For anyone who runs into this issue. Here is the workaround I found. This relates to the answer above. And lets you use registries with ports in the name.
volumes:
- name: certs
secret:
items:
- key: ca.crt
path: my-registry:443/ca.crt
secretName: registry-ca
volumeMounts:
- mountPath: /etc/docker/certs.d
name: certs
readOnly: true
© 2022 - 2024 — McMap. All rights reserved.
insecure-registriesthe preferred approach should be to install the CA correctly on the host. This is a good option though if you just want EKS Managed Node Groups to work with AWS provided AMIs with minimum amount of modifications. Please note for Artifactory you need to provide the full long path to the registry, my-local.artifactory.mydomain.com not just artifactory.mydomain.com – Sello