How do I upload to Google Cloud Storage with Curl and an API key?

Asked 23/2, 2017 at 14:26 Answered 23/2, 2023 at 6:45

Solved http curl post google-cloud-storage

I am trying to upload to google cloud storage with Curl and an API key, without success. The error message seems to indicate that I lack the Content-length header, which I don't. Any ideas?

$ curl -v -T ./myfile -X POST https://storage.googleapis.com/my-app/my-bucket/myfile?key=<my-api-token>
> Host: storage.googleapis.com
> User-Agent: curl/7.51.0
> Accept: */*
> Content-Length: 4
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 411 Length Required
< Date: Thu, 23 Feb 2017 13:46:59 GMT
< Content-Type: text/html; charset=UTF-8
< Server: UploadServer
< Content-Length: 1564
< Alt-Svc: quic=":443"; ma=2592000; v="36,35,34"
< Connection: close
< 
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 411 (Length Required)!!1</title>
  <style>
    [snip snip]
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>411.</b> <ins>That’s an error.</ins>
  <p>POST requests require a <code>Content-length</code> header.  <ins>That’s all we know.</ins>
* Curl_http_done: called premature == 0
* Closing connection 0

Tungstic answered 23/2, 2017 at 14:26 Comment(0)

API keys do not provide authentication. They associate a call with a project for purposes of quota and a few other things, but you cannot use them to access resources that require anything beyond anonymous access.

What you'll need is an "Access Key", which can be acquired in a variety of ways, usually via an OAuth exchange. If you have the gcloud command installed, the easiest way to grab a quick access key to use with cURL is to run gcloud auth print-access-token. The following should work:

$> curl -v --upload-file my-file.txt \
    -H "Authorization: Bearer `gcloud auth print-access-token`" \ 
    'https://storage.googleapis.com/my-bucket/my-file.txt'

Syllogize answered 24/2, 2017 at 18:42 Comment(5)

Thanks! Now I get a different error:

<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>POST object expects Content-Type multipart/form-data</Details></Error>

– Tungstic 1/3, 2017 at 11:20

What curl command are you using now? Looks like you're doing a POST instead of a PUT. – Syllogize 1/3, 2017 at 18:1

That was it! I somehow still had -X POST in the command. Thanks. – Tungstic 2/3, 2017 at 11:24

I could not get this to work with this URL. Instead I needed to use the form: https://{bucket}.storage.googleapis.com/{object} – Onslaught 25/1, 2022 at 17:42

I mean, if we have gcloud to get the auth access token, then it's easier to use gsutil than curl. So... The real problem is that gcloud is a half a gig and I don't want that on my container. – Scandalize 12/3, 2022 at 18:30

Updated Apr 2023 !!!

Check this GCP document out!

TLDR; Follow these steps:

Collect token: use gcloud auth print-access-token or do steps 1-6
Open GCP Oauth Playground
Select permissions under "Cloud Storage API v1"
Click "Authorize APIs"
You will be redirect to your GCP account, allow "Google OAuth 2.0 Playground"
You will redirect to a page with "Authorization code", clicking "Exchange authorization code for tokens"
Collect the value of "access_token" to use later
To upload

TOKEN="your token from step #0 or #6"
curl -X POST -T /path/to/your/file/big_file.json \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    "https://storage.googleapis.com/upload/storage/v1/b/BUCKET_NAME/o?uploadType=media&name=big_file.json"

Note 1: curl which is installed via snap may not work: it may say it's unable to read the file. Reinstall curl with apt install curl

Note 2: if you upload a big file using --data-binary like below, curl may crash due to OOM, use -T instead. --data-binary tries to load the whole file into RAM before sending while -T just streams it.

curl -X POST --data-binary @/path/to/your/file/big_file.json \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
"https://storage.googleapis.com/upload/storage/v1/b/BUCKET_NAME/o?uploadType=media&name=big_file.json"

To download (sometimes you need to mark the object as public)

(curl -X GET -H "Authorization: Bearer $TOKEN" \
-o "/path/to/download/folder/big_file.json" \
"https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/big_file.json?alt=media") >/dev/null

Note: Use >/dev/null to prevent curl from printing out the file content. This is much faster especially when your file is big.

Leitmotiv answered 23/2, 2023 at 6:45 Comment(0)

Recommended topics

Hot tags