AKS. Can't pull image from an acr

Question

I try to pull image from an ACR using a secret and I can't do it.

I created resources using azure cli commands:

az login
az provider register -n Microsoft.Network
az provider register -n Microsoft.Storage
az provider register -n Microsoft.Compute
az provider register -n Microsoft.ContainerService

az group create --name aksGroup --location westeurope

az aks create --resource-group aksGroup --name aksCluster --node-count 1 --generate-ssh-keys -k 1.9.2
az aks get-credentials --resource-group aksGroup --name aksCluster

az acr create --resource-group aksGroup --name aksClusterRegistry --sku Basic --admin-enabled true

After that I logged in and pushed image successfully to created ACR from local machine.

docker login aksclusterregistry.azurecr.io
docker tag jetty aksclusterregistry.azurecr.io/jetty
docker push aksclusterregistry.azurecr.io/jetty

The next step was creating a secret:

kubectl create secret docker-registry secret --docker-server=aksclusterregistry.azurecr.io --docker-username=aksClusterRegistry --docker-password=<Password from tab ACR/Access Keys> [email protected]

And eventually I tried to create pod with image from the ACR:

#pod.yml
apiVersion: v1
kind: Pod
metadata:
  name: jetty
spec:
  containers:
  - name: jetty
    image: aksclusterregistry.azurecr.io/jetty
  imagePullSecrets:
  - name: secret

kubectl create -f pod.yml

In result I have a pod with status ImagePullBackOff:

>kubectl get pods
NAME                    READY     STATUS             RESTARTS   AGE
jetty                   0/1       ImagePullBackOff   0          1m
> kubectl describe pod jetty
Events:
  Type     Reason                 Age              From                               Message
  ----     ------                 ----             ----                               -------
  Normal   Scheduled              2m               default-scheduler                  Successfully assigned jetty to aks-nodepool1-62963605-0
  Normal   SuccessfulMountVolume  2m               kubelet, aks-nodepool1-62963605-0  MountVolume.SetUp succeeded for volume "default-token-w8png"
  Normal   Pulling                2m (x2 over 2m)  kubelet, aks-nodepool1-62963605-0  pulling image "aksclusterregistry.azurecr.io/jetty"
  Warning  Failed                 2m (x2 over 2m)  kubelet, aks-nodepool1-62963605-0  Failed to pull image "aksclusterregistry.azurecr.io/jetty": rpc error: code = Unknown desc = Error response from daemon: Get https://aksclusterregistry.azurecr.io/v2/jetty/manifests/latest: unauthorized: authentication required
  Warning  Failed                 2m (x2 over 2m)  kubelet, aks-nodepool1-62963605-0  Error: ErrImagePull
  Normal   BackOff                2m (x5 over 2m)  kubelet, aks-nodepool1-62963605-0  Back-off pulling image "aksclusterregistry.azurecr.io/jetty"
  Normal   SandboxChanged         2m (x7 over 2m)  kubelet, aks-nodepool1-62963605-0  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed                 2m (x6 over 2m)  kubelet, aks-nodepool1-62963605-0  Error: ImagePullBackOff

What's wrong? Why does approach with secret not work? Please don't advice me approach with service principal, because I would like to understand why this aproach doesn't work. I think it must be working.

Chosen

this looks right and should work. service principal doesnt matter here – Zinkenite

but it doesn't work – Chosen

well, check docker credentials. i highly suggest using lowercare username – Zinkenite

Answer 1

This looks good to me as well. That said, the recommendation is not to use the admin account, rather a service principle. With the SP you gain some granular control over access rights to the ACR instance (read, contributor, owner).

This doc includes two methods for authentication between AKS and ACR using service principles.

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks

Psychomancy

It works. But could you explain please why approach with admin account doesn't work, because I can connect to ACR using admin account from local docker. And how I can get sp password or reset it after creating time. I didn't find this information on portal and I didn't find command which can help me. – Chosen

I have the same problem and too would like to know why this method works and the original does not. – Dyslogistic

Answer 2

The "old" way with AKS was to do create secret as you mentioned. That is no longer recommended.

The "new" way is to attach the container registry. This article explains the "new" way to attach ACR, and also provides a link to the old way to clear up confusion. When you create your cluster, attach with:

az aks create -n myAKSCluster -g myResourceGroup --attach-acr $MYACR

Or if you've already created your cluster, update it with:

az aks update -n myAKSCluster -g myResourceGroup --attach-acr $MYACR

Notes:

• $MYACR is just the name of your registry without the .azurecr.io. Ex: MYACR=foobar not MYACR=foobar.azurecr.io.

• After you attach your ACR, it will take a few minutes for the ImagePullBackOff to transition to Running.

Cliquish

This should be the accepted answer, thanks! – Verdellverderer

Really appreciate for this answer – Muff

I updated and it doesn't work. Did I miss anything here. – Mithridate

Answer 3

I have Azure AKS with Windows node and in my case the error looked like this

Failed to pull image "myacr.azurecr.io/image:tag": [rpc error: code = Unknown desc = failed 
topull and unpack image "myacr.azurecr.io/image:tag": failed to extract layer <digest>: 
Failed to safefile.OpenRelative failed in Win32:
 open \\?\C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots\271
\var\lib\dpkg\info\gcc-12-base:amd64.list: path contains invalid character `:` (0x1f) 
var\lib\dpkg\info\gcc-12-base:amd64.list: unknown, rpc error: code = Unknown desc = 
failed to pull 
and unpack image "myacr.azurecr.io/image:tag": failed to resolve reference
 "myacr.azurecr.io/image:tag": failed to authorize: 
failed to fetch anonymous token: unexpected status: 401 Unauthorized]

And I assumed that the issue was related to authorization. But in fact the problem was that I was trying to pull Linux image into Windows node. After rebuilding image with proper Windows base image the issue was gone.

Horn

Answer 4

It's not exactly the question case. But I was having similar issue with utilization of Attach ACR approach. My problem was with Upper case characters in the registry name. Below warning was being generated by az cli.

Uppercase characters are detected in the registry name. When using its server url in docker commands, to avoid authentication errors, use all lowercase

So ensure to use all lowercases in ACR urls on Docker commands.

Bignonia