Override S3 endpoint using Boto3 configuration file

Question

OVERVIEW:

I'm trying to override certain variables in boto3 using the configuration file (~/aws/confg). In my use case I want to use fakes3 service and send S3 requests to the localhost.

EXAMPLE:

In boto (not boto3), I can create a config in ~/.boto similar to this one:

[s3]
host = localhost
calling_format = boto.s3.connection.OrdinaryCallingFormat

[Boto]
is_secure = False

And the client can successfully pick up desired changes and instead of sending traffic to the real S3 service, it will send it to the localhost.

>>> import boto
>>> boto.connect_s3()
S3Connection:localhost
>>> 

WHAT I TRIED:

I'm trying to achieve a similar result using boto3 library. By looking at the source code I found that I can use ~/aws/config location. I've also found an example config in unittests folder of botocore.

I tried to modify the config to achieve the desired behaviour. But unfortunately, it doesn't work.

Here is the config:

[default]
aws_access_key_id = XXXXXXXXX
aws_secret_access_key = YYYYYYYYYYYYYY
region = us-east-1
is_secure = False
s3 =
    host = localhost

QUESTION:

1. How to overwrite clients variables using config file?
2. Where can I find a complete list of allowed variables for the configuration?

Avidity

May be it's an update but we can do this if we are using python 2: boto.connect_s3( aws_access_key_id=[ACCESS KEY], aws_secret_access_key=[SECRET KEY], host='localhost', port=4572, is_secure=True, calling_format=boto.s3.connection.OrdinaryCallingFormat() ) – Chemulpo

Answer 1

You cannot set host in config file, however you can override it from your code with boto3.

import boto3

session = boto3.session.Session()

s3_client = session.client(
    service_name='s3',
    aws_access_key_id='aaa',
    aws_secret_access_key='bbb',
    endpoint_url='http://localhost',
)

Then you can interact as usual.

print(s3_client.list_buckets())

Mephistopheles

This works great but boto3 is not supported in gsutils boto configuration (cloud.google.com/storage/docs/boto-gsutil). Any alternative in that case? – Esplanade

Answer 2

boto3 only reads the signature version for s3 from that config file. You may want to open a feature request, but for now here is how you can address a custom endpoint:

import boto3
from botocore.utils import fix_s3_host
resource = boto3.resource(service_name='s3', endpoint_url='http://localhost')
resource.meta.client.meta.events.unregister('before-sign.s3', fix_s3_host)

That bit about the meta is important because boto3 automatically changes the endpoint to your_bucket_name.s3.amazonaws.com when it sees fit 1. If you'll be working with both your own host and s3, you may wish to override the functionality rather than removing it altogether.

Thera

how can i do that for sns? – Psychopathy

Answer 3

Another way:

import boto3

s3client = boto3.client('s3', endpoint_url='http://X.X.x.X:8080/',
        aws_access_key_id = 'XXXXXXX',
        aws_secret_access_key = 'XXXXXXXX')
bucket_name = 'aaaaa'
s3client.create_bucket(Bucket=bucket_name)

Engaged

Is the trailing slash and port of the end point url required? – Pawl

You do not need specify port if you hit AWS S3. – Engaged

Isn't that essentially just this answer ? And please do not post an answer that consists essentially of code. Please edit your answer to include an explanation of how and why the code solves the problem, when it should be used, what its limitations are, and if possible a link to relevant documentation. – Ceratoid

Answer 4

using boto3 resource:

import boto3

# use third party object storage
s3 = boto3.resource('s3', endpoint_url='https://URL:443',
  aws_access_key_id = 'AccessKey',
  aws_secret_access_key = 'SecertKey')

# Print out bucket names
for bucket in s3.buckets.all():
 print(bucket.name)

Allain

Answer 5

s3_client = boto3.client(
    "s3",
    "us-east-1",
    aws_access_key_id="test_id",
    aws_secret_access_key="test_key",
    endpoint_url="https://localhost:7000",
    use_ssl=True,
    verify=False,
)

Chord

Isn't that essentially just this answer plus a few keyword arguments ? And please do not post an answer that consists essentially of code. Please edit your answer to include an explanation of how and why the code solves the problem, when it should be used, what its limitations are, and if possible a link to relevant documentation. – Ceratoid

Devil is in the details. This is a complete answer. Those "few keyword arguments" make a difference in whether the code will work. I thought posting a complete answer to supplement a partial answer is a good idea. – Chord

Answer 6

I've just submitted a PR to boto3 to add an env var to allow you to override the endpoint_url if you need to use a dependent module which uses boto3 (in which case you may not be able to modify the call to boto3.client directly)

https://github.com/boto/boto3/pull/2746

https://github.com/rwillmer/boto3

Collision