GCloud Build Failure: ERROR: failed to initialize analyzer, No such object

Question

My most recent builds with GCloud all started failing recently and I am not really sure why. I keep getting 404 errors from the build attempting to find an image that doesn't exist.

Sample error:

Already have image (with digest): us.gcr.io/gae-runtimes/buildpacks/google-gae-22/python/builder:python_20230925_RC00
===> ANALYZING
ERROR: failed to initialize analyzer: getting previous image: getting config file for image "us.gcr.io/growth-ops-apps/app-engine-tmp/app/default/ttl-18h:latest": GET https://storage.googleapis.com/us.artifacts.growth-ops-apps.appspot.com/containers/images/sha256:d13213796512322314e6db634cc57318665191dc5437c121f151cb33310c552e?access_token=REDACTED: unexpected status code 404 Not Found: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.growth-ops-apps.appspot.com/containers/images/sha256:d13213796512322314e6db634cc57318665191dc5437c121f151cb33310c552e</Details></Error>

I can verify that the bucket us.artifacts.growth-ops-apps.appspot.com is empty in the console, but I have no idea why. There is no bucket retention policy or anything I can see that is obvious to me.

For reference, I have a single GitHub repository set to trigger builds to 2 separate projects, one for development and one for production. Only the production one is failing in this way.

I am using a cloudbuild.yaml file stored in the repository.

Any ideas?

Radke

Answer 1

I figured it out! In thinking through one of my own comments, the build is looking for an artifact that doesn't exist, I figured there must be some way of forcing a new build. I found the --no-cache flag in the docs, which forced a brand new build instead of attempting to use information from a non-existent prior build.

Radke

The --no-cache flag did the job. In my case the error appeared because the storage configuration I have, that drops builds after a few days. – Pilocarpine

Is there an equivalent --no-cache flag for gcloud functions deploy? – Zachariahzacharias

This worked for me too – Static

Answer 2

The reason you're seeing the empty bucket is that us.artifacts.<project id>.appspot.com bucket is automatically created and used by Cloud Build to store container images. Also, every time you deploy a new version in App Engine, it stores the object in this bucket.

Try using the Lifecycle Object Management to delete the objects in the bucket after a certain period of time.

Reference:

• Google creates a storage bucket without my interaction

Edit:

A solution that you can do here is to first remove the lifecycle configuration. Then you can retry the whole build process by adding the image again.

Darkling

Ok so upon further research, it looks like I did have Lifecycle rules for that bucket and the artifact was indeed deleted. I didn't have versioning enabled at the time, so I can't recover anything. Any advice on how to get the deployment to succeed now though? – Radke

I have edited my answer; you may want to check it out. Let me know if it helped. – Darkling

+1, seeing the same with our pipeline. @Marra are you saying to remove the bucket lifecycle rule and kick off another deployment? Will we not see the same issue trying to pull the previous image? – Taegu

So that is the crux of my problem though. I deleted the lifecycle rules, there are no existing artifacts, and yet the build is looking for an artifact that doesn't exist, so it continues to fail even after implementing your suggestion. – Radke

Answer 3

You need to migrate to Artifact-registry

[https://cloud.google.com/artifact-registry/docs/transition/setup-gcr-repo][1]

Mikemikel

Answer 4

The accepted answer doesn't work for Cloud Functions, as there is no --no-cache flag. Google's own troubleshooting link says to do the following:

1. Disable Lifecycle Management on the buckets required by Container Registry.
2. Delete all the images of affected functions. You can access build logs to find the image paths. Reference script to bulk delete the images. Note that this does not affect the functions that are currently deployed.
3. Redeploy the functions.

However, there appears to be a bug as multiple of us have followed these steps and are still getting the build error. The only solution that worked for me was to migrate to Artifact Registry per the answer from @calle-engene

Zachariahzacharias

Answer 5

I ran into this error too. Go to container registry (in your case us.gcr.io/growth-ops-apps/app-engine-tmp/app/default/ttl-18h), delete all of the images inside that folder, and try gcloud app deploy again. It worked for me.

Irrevocable