Python 3 Boto 3, AWS S3: Get object URL

Question

I need to retrieve an public object URL directly after uploading a file, this to be able to store it in a database. This is my upload code:

   s3 = boto3.resource('s3')
   s3bucket.upload_file(filepath, objectname, ExtraArgs={'StorageClass': 'STANDARD_IA'})

I am not looking for a presigned URL, just the URL that always will be publicly accessable over https.

Any help appreciated.

Write

Answer 1

There's no simple way but you can construct the URL from the region where the bucket is located (get_bucket_location), the bucket name and the storage key:

bucket_name = "my-aws-bucket"
key = "upload-file"

s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
bucket.upload_file("upload.txt", key)
location = boto3.client('s3').get_bucket_location(Bucket=bucket_name)['LocationConstraint']
url = "https://s3-%s.amazonaws.com/%s/%s" % (location, bucket_name, key)

Cloudy

This answer is wrong, you're not quoting the key properly. Also fails for new buckets that must be addressed using a domain name. – Connected

Now URL is changed to f"{bucket_name}.s3.{location}.amazonaws.com/{key}" – Pickar

Answer 2

Since 2010 you can use a virtual-hosted style S3 url, i.e. no need to mess with region specific urls:

url = f'https://{bucket}.s3.amazonaws.com/{key}'

With quoted key :

url = f'''https://{bucket}.s3.amazonaws.com/{urllib.parse.quote(key, safe="~()*!.'")}'''

Moreover, support for the path-style model (region specific urls) continues for buckets created on or before September 30, 2020. Buckets created after that date must be referenced using the virtual-hosted model.

See also this blog post.

Steve

This does not work for GovCloud (and I assume for China) – Sponger

Can you please explain a bit more ? (For china it's amazonaws.com.cn I guess) – Steve

This relies on the s3.amazonaws.com endpoint figuring out which region your bucket is in. That works for the aws partition, but it doesn't know anything about the GovCloud and China partitions. For GovCloud, it looks like s3-us-gov-west-1.amazonaws.com. – Sponger

They should have this implemented in the api. It makes no sense to expect people to dig into details of which server at which time expects which format – Weiser

Doesn't quote key properly. – Connected

If your key contains spaces, S3 replaces those with '+'. A better approximation might be to replace quote with quote_plus in the above. – Schlicher

As described in another answer, you can use boto3 to generate it by calling generate_presigned_url() with a Config(signature_version=botocore.UNSIGNED) which will generate the expected URL (without expiration nor signature), like https://my-bucket.s3.amazonaws.com/my_file. – Regine

Answer 3

Concatenating the the raw key will fail for some special characters in the key(ex: '+'), you have to quote them:

url = "https://s3-%s.amazonaws.com/%s/%s" % (
    location,
    bucket_name,
    urllib.parse.quote(key, safe="~()*!.'"),
)

Or you can call:

my_config = Config(signature_version = botocore.UNSIGNED)
url = boto3.client("s3", config=my_config).generate_presigned_url(
    "get_object", ExpiresIn=0, Params={"Bucket": bucket_name, "Key": key}
)

...as described here.

Caracara

Please note that https://s3-us-east-1... does not work, other regions do. To have the same behaviour you should use https://s3.us-east-1... (note the . instead of - ) See docs.aws.amazon.com/general/latest/gr/s3.html – Unpack

ExpiresIn=0 is not required. – Regine

Answer 4

You can generate a presigned URL and then trim its query parameters. This requires the "s3:PutObject" permission for the relevant bucket.

url = s3client.generate_presigned_url(ClientMethod = 'put_object',
                                      Params = { 'Bucket': bucket_name, 'Key': key })

# trim query params
url = url[0 : url.index('?')]

Alcoholicity

Answer 5

You should use the event system, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#request-created

Natation

Answer 6

Just a small note. The function call

location = 
    boto3.client('s3').get_bucket_location(Bucket=bucket_name['LocationConstraint']

may return location = None if the bucket is in the region 'us-east-1'. Therefore, I'd amend the above answer and add a line below that line:

if location == None: location = 'us-east-1'

Variegation