gsutil cannot copy to s3 due to authentication

Question

I need to copy many (1000+) files to s3 from GCS to leverage an AWS lambda function. I have edited ~/.boto.cfg and commented out the 2 aws authentication parameters but a simple gsutil ls s3://mybucket fails from either an GCE or EC2 VM.

Error is The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256..

I use gsutil version: 4.28 and locations of GCS and S3 bucket are respectively US-CENTRAL1 and US East (Ohio) - in case this is relevant.

I am clueless as the AWS key is valid and I enabled http/https. Downloading from GCS and uploading to S3 using my laptop's Cyberduck is impracticable (>230Gb)

Burrus

Answer 1

As per https://issuetracker.google.com/issues/62161892, gsutil v4.28 does support AWS v4 signatures by adding to ~/.boto a new [s3] section like

[s3]
# Note that we specify region as part of the host, as mentioned in the AWS docs:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
host = s3.eu-east-2.amazonaws.com
use-sigv4 = True

The use of that section is inherited from boto3 but is currently not created by gsutil config so it needs to be added explicitly for the target endpoint.

For s3-to-GCS, I will consider the more server-less Storage Transfer Service API.

Burrus

gsutil does support this as of version 4.28 -- the issue you linked just didn't get updated until now, apologies :) Please consider changing your answer to reflect this. – Metempsychosis

@Metempsychosis thanks for the tip ! Answer and issuetracker updated. – Burrus

Answer 2

I had a similar problem. Here is what I ended up doing on a GCE machine:

Step 1: Using gsutil, I copied files from GCS to my GCE hard drive Step 2: Using aws cli (aws s3 cp ...), I copied files from GCE hard drive to s3 bucket

The above methodology has worked reliably for me. I tried using gsutil rsync but it fail unexpectedly.

Hope this helps

Negrillo

Thanks - ended up doing that for a one-time transfer but I need to do the same the other way round to load into BigQuery + eventually this will happen daily. Copying locally is a workaround but still believe there is an issue with gsutil – Burrus