GCP Cloud Function - ERROR fetching storage source during build/deploy

Question

Running into problems building deploying functions. When trying to programmatically deploy the function I get the following output in builder logs (ERRORS).

2020-10-20T02:22:12.155866856Z starting build "1fc13f51-28b6-4052-9a79-d5d0bef9ed5c"
 I 
2020-10-20T02:22:12.156015831Z FETCHSOURCE I 
2020-10-20T02:22:12.156031384Z Fetching storage object: gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655 I 
2020-10-20T02:22:16.698838385Z Copying gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655... I 
2020-10-20T02:22:16.848908949Z / [0 files][    0.0 B/  7.6 KiB]                                                
/ [1 files][  7.6 KiB/  7.6 KiB]                                                
 I 
2020-10-20T02:22:16.848965618Z Operation completed over 1 objects/7.6 KiB.                                       I 
2020-10-20T02:22:17.848654342Z replace /workspace/pb/__init__.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL I 
2020-10-20T02:22:17.848694354Z (EOF or read error, treating as "[N]one" ...) I 
2020-10-20T02:22:17.848697209Z Archive:  /tmp/source-archive.zip I 
2020-10-20T02:22:17.848698273Z   inflating: /workspace/main.py       I 
2020-10-20T02:22:17.848699274Z    creating: /workspace/pb/ I 
2020-10-20T02:22:17.848700378Z   inflating: /workspace/pb/__init__.py   I 
2020-10-20T02:22:17.848701376Z   inflating: /workspace/pb/${FILENAME}.py   I 
2020-10-20T02:22:17.848702578Z   inflating: /workspace/requirements.txt   I 
2020-10-20T02:22:18.477862579Z Fetching storage object: gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655 I 
2020-10-20T02:22:21.795484700Z Copying gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655... I 
2020-10-20T02:22:21.937040057Z / [0 files][    0.0 B/  7.6 KiB]                                                
/ [1 files][  7.6 KiB/  7.6 KiB]                                                
 I 
2020-10-20T02:22:21.937094331Z Operation completed over 1 objects/7.6 KiB.                                       I 
2020-10-20T02:22:22.934311112Z replace /workspace/main.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL I 
2020-10-20T02:22:22.934315945Z (EOF or read error, treating as "[N]one" ...) I 
2020-10-20T02:22:22.934318352Z Archive:  /tmp/source-archive.zip I 
2020-10-20T02:22:23.569955715Z Fetching storage object: gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655 I 
2020-10-20T02:22:26.802351633Z Copying gs://gcf-sources-629360234120-us-central1/${FUNCTIONNAME}-63f501f1-a8d2-4837-b992-1173ced83036/version-1/function-source.zip#1603160527600655... I 
2020-10-20T02:22:26.867398009Z / [0 files][    0.0 B/  7.6 KiB]                                                
/ [1 files][  7.6 KiB/  7.6 KiB]                                                
 I 
2020-10-20T02:22:26.867408400Z Operation completed over 1 objects/7.6 KiB.                                       I 
2020-10-20T02:22:27.863532350Z replace /workspace/main.py? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL I 
2020-10-20T02:22:27.863536996Z (EOF or read error, treating as "[N]one" ...) I 
2020-10-20T02:22:27.863538973Z Archive:  /tmp/source-archive.zip I 
2020-10-20T02:22:27.995306558Z ERROR I 
2020-10-20T02:22:27.995310537Z ERROR: error fetching storage source: generic::unknown: retry budget exhausted (3 attempts): fetching gcs source: unpacking source from gcs: source fetch container exited with non-zero status: 1 I 

Confusingly, if I deploy the function from the UI, it builds/deploys fine. That makes me think this is some sort of IAM/permissions around the builder/agent. But, am rather lost. Any tips/thoughts? Thanks!

Jointress

Hi @Jointress indeed, seems to be a permission issue, considering that via UI you are able to. When deploying via command, it's usually the service account the agent that will be building and deploying the application. Considering that, could you please take a look at this documentation here and get more details on service accounts? This way, you will be able to check those that are being used when you are deploying your application and then, confirm if they have the same permissions as your account. – Consumerism

Thanks, @Consumerism -- There is indeed something special with the service accounts. And, I'm deploying everything else via terraform. I had worked around via just deploying the specific resources via gcloud command in extra deploy steps... But, yes, I should take the time to sort out the permissions. – Jointress

It has been awhile -- but, IIRC (from many months back), it was the permissions for using Cloud Build to be able to build/deploy the function. – Jointress

Answer 1

Had the same error message after spending a day troubleshooting I finally found a way.

I noticed I had node@15 installed on my PC and node@12 for firebase function.

All I need do was to

• install node@12 on my PC
• unlink the previous node (i.e node@15) and
• link to the newly installed node@12.

Boom! it uploaded.

Doorman

This answer should be accepted. I changed my Node@15 to Node@12. Unlink node15, link node12 then it works perfectly. – Permute

@TruongHieu I'm glad it helped :-) – Doorman

I want to add that I use Firebase functions, and it worked for me also. Yeah!!! Thank you :) – Sarcophagus

so you can downgrade your version with n with LTS version $ npm install -g n $ n lts – Gameness

Amazing, I never would have figured that out myself! – Rex

I really wonder how this cloud builder error can be the result of the local NodeJs version. If the firebase / gcloud CLI has this kind of dependency they should really enforce the Node version in the CLI. – Drawknife

Answer 2

I had the same problem deploying cloud functions on my newly started Firebase projects. My local version of node was 15

For beginners (like me) which were struggling downgrading node on Mac, you can use the following commands:

sudo npm install -g n
sudo n 12

Krawczyk

Answer 3

As Prodigy mentioned, something must has broken the Cloud Functions deployment in the latest Node v15.6.0 (or another recent one).

Has the same issue today with firebase deploy. Resolved by rolling back to Node 12 (and I am sure any other more recent version works great).

Disapproval

I had exactelly the same issue and your solution helped me. Thanks – Proverbs

Answer 4

I can confirm the issue came with node update to 15.6.0. Running Manjaro, updated my OS, rolling back to node 12 (with nvm) solved the issue!

Switchboard

Likewise. I had this failing on a Gitlab build which was using node:latest as the container. Turned out it was 15.6.0. I moved down to node:10.23.1 and it worked. – Torrlow

Answer 5

Even downgrade nodejs to version 15.5.1 works for me. GC function definitely don't like version 15.6.0

Stellite

this was the solution for me!!! I ended up removing node via homebrew with brew unlink node and brew uninstall node and then installed n via homebrew brew install n and then install v15.6.0 with n 15.6.0 – Earvin

Answer 6

It has nothing to do with permissions. It must be something else. Maybe google sets limits as to how big cloud function should be.

Ampersand

Answer 7

I had a similar issue today. It turned out that on one function I was using a timeoutSeconds that exceeded the max of 540 as specified here.

Latashalatashia

Answer 8

Update for 3/2022, firebase deploy no longer supports node@12. I had node@18 installed, and by setting

"engines": {
    "node": "18"
  }

in package.json I was able to fix this issue

Muir