Error: ImagePullBackOff and Error: ErrImagePull errors with GKE

Question

I'm getting the following error when kubectl apply a .yaml to deploy a custom built docker image to our cluster in GCP (edited out sensitive info):

Have tried the following but no luck:

• manually deploying the image
• checking to ensure there is firewall rules to allow 443 and there is, nothing blocking it either
• tried setting container registry to public
• checked service account perms and that the secrets for authorizing the pull is there
• verified the pull works on my end using the url given in the above image
• tried destroying whole infra. and respinning up
• ensured control planes are like for like as our other environment
• in our terraform, the cluster and node pools have all the required 0authscopes

Weird thing is, we have a dev. environment which is the exact same (using terraform to spin up the infra) and using the same files + images (obviously different projects so required things have been changed to match that) and no issues.

Crosscut

Is your cluster private by any chance (aka the nodes doesn't have a public IP) ? And do you have Google Private Access enabled on the VPC ? The error seems to be that the node is not able to pull to pull the image from GCR because it cannot reach it. This happens when A) The cluster has private nodes, B) There is no Cloud NAT for the region of that cluster and C) you don't have Private Access enabled on the subnet/vpc – Ravish

Thanks, solution was "you don't have Private Access enabled on the subnet/vpc", enabling this fixed it – Crosscut

Great i will post it as an answer. if you could accept it it would be great. – Ravish

Answer 1

This happen usually when your cluster:

• has private nodes (aka no Public IP's)
• There is no Cloud NAT for the region of that cluster
• You don't have Private Access enabled on the subnet/vpc

Basically the error is that the node is not able to pull the image from the registry. You might have two cases:

• If your images are hosted in GCR or AR (Google Container Registry or Artifact Registry). Even if your nodes are private, Google Private Access must be enabled to allow the node to reach the registry
• If your images are hosted on an external registry (Exp: Docker). You have to have either nodes with a public IP or Cloud NAT to reach the internet
• The third case would be if you have self-hosted registries onPrem or on an other Cloud. You need some sort of private connection between GCP and where the registry is hosted (VPN, Interconnect...) and you need to configure Cloud DNS to resolve the registry to the private IP

Ravish

Answer 2

I had this issue when trying to migrate from GCR to GAR (due to GCR deprecation).

What resolved this for me is giving the GKE default service account ([email protected]) the Artifact Registry Reader Role. This can be done in the IAM & Admin -> IAM -> GRANT ACCESS.

If you're not using the default service account, to find out which service account should get the role, navigate to:
GKE -> Clusters -> Select your cluster -> Nodes tab -> Select the node pool -> Scroll down to Security section Service account

Bodleian