Push to google container registry fails: Retrying

Question

I'm trying to push to the Google container registry from my Jenkins. The builds run inside the Kubernetes Jenkins Plugin, which uses the gcr.io/cloud-solutions-images/jenkins-k8s-slave to build the docker image into the Kubernetes native Docker.

After authenticating to the Google container registry I'm trying to push the newly built image. This is my pipeline script:

def imageTag = 'gcr.io/project-id/tag'

def version = version from pom 

sh './mvnw package'

sh "docker build -t $imageTag:$version ."

sh('gcloud auth activate-service-account --key-file=$FILE')

sh('docker login -p $(gcloud auth print-access-token) -u _token https://gcr.io')

sh("gcloud docker -- push $imageTag:$version")

The push fails with the following output:

c6ff94654483: Preparing
209db64c273a: Preparing
762429e05518: Preparing
2be465c0fdf6: Preparing
5bef08742407: Preparing
c6ff94654483: Retrying in 5 seconds
5bef08742407: Retrying in 5 seconds
209db64c273a: Retrying in 5 seconds
2be465c0fdf6: Layer already exists
762429e05518: Layer already exists
c6ff94654483: Retrying in 4 seconds
5bef08742407: Retrying in 4 seconds
209db64c273a: Retrying in 4 seconds
c6ff94654483: Retrying in 3 seconds
5bef08742407: Retrying in 3 seconds
209db64c273a: Retrying in 3 seconds
c6ff94654483: Retrying in 2 seconds
5bef08742407: Retrying in 2 seconds
209db64c273a: Retrying in 2 seconds
c6ff94654483: Retrying in 1 second
5bef08742407: Retrying in 1 second
209db64c273a: Retrying in 1 second
5bef08742407: Retrying in 10 seconds
...
unexpected EOF

Leach

I've been experiencing the same issue the past few days attempting to setup Gitlab's CI/CD runner to automatically login to GCR when pushing/pulling images to the private repositories. What user account are you logged in with? A personal one, or a service account? Does it have the appropriate IAM permissions? – Pandiculation

I'm using a service account with it's json file and didn't modify it's permissions. Maybe it doesn't have write permissions to the bucket. I'll check that later and let you know. – Leach

I just read through your pipeline. Is there a reason you're passing _token as the username? The advanced authentication page seems to say to use oauth2accesstoken? cloud.google.com/container-registry/docs/… – Pandiculation

Feel free to reach out to the GCR team directly. Contact info here: cloud.google.com/container-registry/docs/support/… – Pangolin

Answer 1

The root cause of this issue is that your docker daemon is not authenticated with the credentials necessary to push to gcr.io. For the original question, I believe this is likely because the user account being used was _token instead of oauth2accesstoken.

I was experiencing an error similar to this, except that instead of using docker login, I was using docker-credential-gcr and was getting the same unexpected EOF error.

My problem was the fact that I was running on GCE, from which docker-credential-gcr was detecting and using a different service account via the GCE metadata API.

So, for others experiencing this issue who are running on GCP and trying to authenticate a service account via docker-credential-gcr, you need to tell it to only look at the gcloud credentials, instead of looking at the environment for the metadata API details. My flow looks like this now:

gcloud auth activate-service-account --key-file=$FILE

docker-credential-gcr configure-docker --token-source="gcloud"

docker push gcr.io/....

Hope it helps someone.

Pandiculation

Answer 2

In case it helps anyone: I was hitting this when pushing to Google Artifact Registry.

I expected Artifact Registry to function just like GCR, but it turns out that a repository in Artifact Registry can contain many different images; so I needed to add a segment in my image name i.e.

docker push <host>/<account>/<GAR repository>/<image repo>:tag

I figured this out by looking at the Docker Daemon logs and the answer to this question

Awry

I was making the same mistake. After waiting a minute or so while it showed Retrying in 1 second, I got this error, which helped me identify the issue: name invalid: Missing image name. Pushes should be of the form docker push HOST-NAME/PROJECT-ID/REPOSITORY/IMAGE – Grandiloquence

Answer 3

Check if you use correct projectID in tag as it was solved in Cannot push image to repository in Google Container Engine

Jeffreyjeffreys

Thanks! This was my case! – Marsala

Answer 4

For people having issue pushing builds to GCR, my mistake was that I was using project name instead of project id.

Thanks to https://forums.docker.com/t/cannot-push-image-to-repository-in-google-container-engine/12662 for helping me sort this

Melamie

Answer 5

In my case, I observed a similar 'retrying' problem when trying to push to GCR in various ways, having installed Jenkins on GKE per Google Cloud Services packaged tutorial.

I used the default service account for the slaves that were having this problem. This inherits the GCE cluster OAuth scopes, by default these don't have write permissions for Cloud Storage. The Google Cloud console shows this under Permissions for the Kubernetes cluster. It showed Storage: Read Only, and unfortunately it can't be changed.

I ended up adding a fresh node pool as described in this excellent article, then removing the original node pool. The create command looks like gcloud container node-pools create pool-3 --cluster my-cluster --zone europe-west1-b --num-nodes=3 --scopes https://www.googleapis.com/auth/devstorage.read_write --machine-type g1-small

After doing this, the push worked, and the Permissions list on the GKE cluster showed Storage: Read Write.

Baluster

Answer 6

Alternatively, docker login with keyfile and push to registry can also be done via following command:

 docker login -u _json_key --password-stdin https://eu.gcr.io < $FILE
 docker push eu.gcr.io/<PROJECT_ID>/<IMAGE_NAME>:<VERSION>

This worked for me.

Stott

Answer 7

Execute this command and update config.json in docker

gcloud auth configure-docker asia-docker.pkg.dev

Alfalfa

Answer 8

In case this helps anyone else ...

There should be no hyphen between east-1

Check you're using the correct, region. I was doing:

docker push us-east-1-docker.pkg.dev/...

Instead of

docker push us-east1-docker.pkg.dev/...

Road

Answer 9

I was stuck on the same error loop while using the following command to build the repo -

gcloud builds submit -t europe-west1-docker.pkg.dev/PROJECT_NAME/container .

So I tried following steps to push the container image.

1. gcloud artifacts repositories create container --repository-format=docker --location=europe-west1

2. gcloud builds submit -t europe-west1-docker.pkg.dev/<PROJECT_NAME>/container/my_image:v1 .

And then the image was pushed successfully.

Basically in ARTIFACT REGISTRY we can have multiple container images.

Rossie

Answer 10

please check whether

sh "docker build --no-cache -t $imageTag:$version ."

solves it

Disciplinarian

Unfortunately it's still the same issue. The build works fine. – Leach