How do I upload a file size greater than 30MB to google cloud storage with gsutil?

Question

I can't seem to find the documentation about how to upload files that are greater than 30MB to google cloud storage. I have just been using:

gsutil cp test.csv gs://example

Any help would be awesome. Thanks

Telecast

You should be able to upload bigger files with gsutil. In theory it inherits the limits of Google Cloud Storage and this one "supports objects that can be terabytes in size"[1]. Could you run the following command?: gsutil -D cp test.csv gs://example That would give you more details. I'd like you to share with us that information (you can upload it to pastebin for example). If so, please remember to remove all the confidential data that appears in the output of that command. [1] cloud.google.com/storage/docs/overview#features – Naturalism

Uploading files that large should work. What is the problem you're seeing when you try to upload files greater than 30MB? – Lauren

I have uploaded many huge files with the same command and it works fine. gsutil -m cp -r source destination this is the command I am using. -m option splits it into multiple processes to make it fast. Can you put error logs for details? – Chaos

Answer 1

If you are using the CSV format you can also upload the gzip version. Just create a gzip file and upload on GCS using below:

gsutil cp <localgzipfile> gs://location/of/gzipfile.csv.gz

I am using this strategy to upload more than 200MB flat file whose gzip version is mere 40 MB. Hope this helps.

More info can be found here

Meryl