Downloading files from AWS S3 Bucket with boto3 results in ClientError: An error occurred (403): Forbidden

Question

I am trying to download files from a s3 bucket by using the Access Key ID and Secret Access Key provided by https://db.humanconnectome.org. However, even though I am able to navigate the database and find the files (as I have configured my credentials via aws cli), attempting to download them results in the following error: "botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden"

With the same credentials, I can browse the same database and download the files manually via a cloud storage browser such as Cyberduck, so how Cyberduck accesses the data does not invoke a 403 Forbidden error.

I have also verified that boto3 is able to access my aws credentials, and also tried by hardcoding them.

How I am attempting to download the data is very straightforward, and replicates the boto3 docs example: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html

s3 = boto3.client('s3',
    aws_access_key_id=ACCESS_KEY_ID,
    aws_secret_access_key=ACCESS_KEY,)

s3.download_file(Bucket=BUCKET_NAME, Key=FILE_KEY, Filename=FILE_NAME)

This should download the file to the location and file given by FILE_NAME, but instead invokes the 403 Forbidden error.

Slender

Try to make sure that you have the proper s3 bucket name and the correct file key. According to docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html If you do not have the ListObject permission on that bucket you will get 403 (access denied) instead of 404 ("no such key"). So the error code can be a bit misleading. – Businessman

Answer 1

You'll need to pass the bucket region as well when downloading the file. Try configuring region using the CLI or pass region_name when creating the client.

s3 = boto3.client('s3',
    aws_access_key_id=ACCESS_KEY_ID,
    aws_secret_access_key=ACCESS_KEY,
    region_name=AWS_REGION)

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html

Peradventure

Still does not work despite specifying the region in my case :( – Gar

Same. Works from my laptop, but not from an aws batch job – Tellurion

Answer 2

I know this might sound ridiculous, but make sure you don't have a typo in your bucket name or anything like that.

I worked so long trying to fix this, only to realize I added an extra letter in the env variable I had set for my s3 bucket.

It's weird that they give you a forbidden error as a opposed to "not found" error, but they do.

Wilmington