django-storages with multiple S3 Buckets

Question

I am using AWS and I will have different buckets in my application. I am also using Django-Storages. Is there a way to specify which bucket I want to upload the file to (for example, as a parameter in the Save() function or whatever?)

I saw this Django - Error importing storages.backends, but I don't understand how it should be used?!

Munmro

Answer 1

S3BotoStorage takes the bucket name as a parameter. If not given it will use the AWS_STORAGE_BUCKET_NAME setting. That means if you want to make S3BotoStorage the default storage backend with DEFAULT_FILE_STORAGE then it must use the default bucket.

However you can also assign a storage on a field level:

from django.db import models
from storages.backends.s3boto import S3BotoStorage

class MyModel(models.Model):
    file_1 = models.FileField() # Uses default storage
    file_2 = models.FileField(storage=S3BotoStorage(bucket='other-bucket'))

Edit:

Comments are getting out of hand so I'll update my answer. Changing the parameters of the storage backend on an instance basis is not something that the Django storage API was designed to do. The storage backend does not have knowledge of the model instance because the storages can be used outside the context of a model such as with static files. Not completely unreasonable but it's not a usage that Django or django-storages was intended to solve. I don't expect you aren to find a drop in storage backend that will handle this for you.

The docs describe how you can manage files manually: https://docs.djangoproject.com/en/1.9/topics/files/#storage-objects At a minimum you would need store the bucket where you saved the file somewhere so that you can find it later when you query the model.

Irresoluble

Nice... so I don't need to overwrite as shown here? #9431270 I can just pass the bucket like you just showed me?! – Munmro

Yes for this use-case you do not since you can pass the bucket name. – Irresoluble

Quick question Mark: In the answer, you are passing the bucket name in static. Any chances we can pass that dynamically? As the bucket name will be resolved at runtime... – Munmro

No not in the model definition. For that you would need a subclass of S3BotoStorage and possibly FileField depending on how you intend to resolve the bucket name at runtime. – Irresoluble

What if I want to do for example MyObject.save(bucket_name) where bucket_name is a variable... How would I do that?! – Munmro

My app will have multiple clients, each part of a certain company. I would like for each company to have his own bucket, but since they will all be talking to 1 Django Server, I need some way of spreading the content to their respective bucket... Or what I am trying to do is completely stupid? – Munmro

@Munmro I once modified django-cumulus to add some addition syntax to is url schema. In short I made it possible to change the rackspace "bucket" by appending bucket-name: to the uploaded string, and django-cumulus was designed split the front of the folder name/URL using the : character. Sorry if thats a tad verbose, look at: github.com/thomaspurchas/django-cumulus – Catlike

Update from 2019. Use S3Boto3Storage instead S3BotoStorage – Kunming

At some point this parameter appears to have changed to bucket_name, and using bucket will return an error: django.core.exceptions.ImproperlyConfigured: Invalid setting 'bucket' for S3Boto3Storage. See django-storages source: github.com/jschneier/django-storages/blob/…. – Martinemartineau

Answer 2

Another solution if you want to specify bucket on the runtime, you can do so before invoking the save() method on the model.

Following the above example:

from django.db import models
from storages.backends.s3boto import S3BotoStorage

class MyModel(models.Model):
    file_1 = models.FileField() # Uses default storage
    file_2 = models.FileField()

In views when saving the model, you can specify the storage on that field.

my_file_model = MyModel()
my_file_model.file_2.storage = S3BotoStorage(bucket="your-bucket-name")
my_file_model.save()

In this way file_2 will be saved in the bucket you specify where file_1 will use your default bucket.

Then

This is cool but be careful to assign your file BEFORE you assign your new storage system. Otherwise Default storage will override your custom one. – Ivatts

I am sorry if this is a silly question, I am a new bee with django. How do you recommend writing the view for above example, can't we simply save that in the model? – Maceio

I used this method and overwrite save(): def save(self, *args, **kwargs): self.file_field.storage = MyS3Boto3Storage(bucket_name=self.bucket_name()) super().save(*args, **kwargs) self.bucket_name() - function that get bucket name depends on model options. – Junket

Answer 3

Just mention another bucket name in settings.py with PRIVATE_BUCKET_NAME='bucket name'.

Create a custom class which override S3BotoStorage and which can be serialize into migration files.

Create a object for class s3_storage = S3MediaStorage() and give it to storage in file1 field in MyModel

from storages.backends.s3boto import S3BotoStorage
from django.conf import settings

@deconstructible
class S3MediaStorage(S3BotoStorage):
    def __init__(self, *args, **kwargs):
        kwargs['bucket'] = getattr(settings, 'PRIVATE_BUCKET_NAME')
        super(S3MediaStorage, self).__init__(*args, **kwargs)

s3_storage = S3MediaStorage()

class MyModel(models.Model):
    file = models.FileField()
    file1 = models.FileField(storage=s3_storage)

Wheezy

deconstructible is located at django.utils.deconstruct. – Companion

Answer 4

For S3Boto3Storage use this:

media_file = models.FileField('media file', storage=S3Boto3Storage(bucket_name='media_bucket'), upload_to='media', blank=True)

Trichosis