Django-Storages with SFTP: GET-requests fail

Question

I am trying to use django-storages to access my "Hetzner" Storage Box (https://www.hetzner.com/storage/storage-box) using SFTP which should hold media data, i.e. image files which users of my website can upload dynamically.

The corresponding part of my settings.py file looks like:

DEFAULT_FILE_STORAGE = 'storages.backends.sftpstorage.SFTPStorage'
SFTP_STORAGE_HOST = 'username.your-storagebox.de'
SFTP_STORAGE_ROOT = '/media'

SFTP_STORAGE_PARAMS = {
'username': 'username',
'password': 'password',
'allow_agent': False,
'look_for_keys': False,
}

The strange thing is that, when the user uploads an Image, it is placed on the storage space as I can confirm using SFTP. But getting the images from the storage box fails, no Image is displayed. An excerpt from the console:

[03/Sep/2021 22:34:01] "GET /media/filename.jpg HTTP/1.1" 404 1962

I could figure out that Django is still looking inside my MEDIA_DIR for the files. Againg, the corresponding part of my settings:

MEDIA_DIR = 'media'
MEDIA_ROOT = os.path.join(BASE_DIR, MEDIA_DIR)
MEDIA_URL = '/media/'

So in a nutshell: Using SFTP seems to be working for putting files into storage, but getting them again somehow fails.

EDIT: As requested, I am going to provide some more code snippets: models.py:

class SizeRestrictedImageField(ImageField):

    def __init__(self, *args, **kwargs):
        self.max_upload_size = kwargs.pop('max_upload_size', 0)
        super().__init__(*args, **kwargs)

    def clean(self, *args, **kwargs):
        data = super().clean(*args, **kwargs)

        file = data.file
        try:
            if file.size > self.max_upload_size:
                raise forms.ValidationError(_('Please keep filesize under %s. Current filesize %s'
                        ) % (filesizeformat(self.max_upload_size),
                        filesizeformat(file.size)))
        except AttributeError:
            logger.exception('An Exception occured while checking for max size of image upload. size: `%s`'
                             , file.size)
            pass

        return data


class ImageModel(models.Model):
    image = SizeRestrictedImageField(upload_to=POST_PIC_FOLDER, null=True, blank=True,
                              help_text="Erlaubte Dateitypen: .jpeg, .jpg, .png, .gif", max_upload_size=MAX_IMAGE_SIZE)

And my urls.py:

urlpatterns = [
                  path('defaultsite/', defaultsite_view, name='home'),
                  path('help', help_view, name="help"),
                  path('user/', include('user.urls')),
                  path('sowi/', include('sowi.urls')),
                  path('blog/', include('blog.urls')),
                  path('chat/', include('chat.urls')),
                  path('notifications/', include('notifications.urls')),
                  path('cookies/', include('cookie_consent.urls')),
                  path('', home_view, name="home"),
                  path('about/', AboutUsView.as_view(), name="about-us"),
                  path('impressum/', impressum_view, name="imprint"),
                  path('privacy/', privacy_view, name='privacy'),
                  path('privacy/statement/', privacy_statement_view, name='privacy-statement'),
                  path('agb', agb_view, name="agb")
              ] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) + static(settings.MEDIA_URL,
                                                                                           document_root=settings.MEDIA_ROOT)

I tried removing the +static(...)-part from my url-patterns, but that didn't seem to solve the issue.

Bridwell

Hi Sarius, I feel having some snippet of the code for the get would be helpful. – Mellins

Can u please add project.urls code of URL patterns in the question? – Sommer

@FaisalNazik Yes, I just did that – Bridwell

@Sarius, I might have another idea based on this issue (github.com/jschneier/django-storages/issues/1045). I don't think you can access your file the same way you used to. GET /media/file.jpg is going to redirect you to the media directory indeed. You need to create a proxy to the sftp to get the file. I believe you might find the So post usefull #50619355 – Mellins

thanks @Paulo. I will try it and if it works, i'll tell you. – Bridwell

@Mellins Do you have an idea if there is any way of implementing this without using the REST-Framework? – Bridwell

Would you consider ditching sftp, for an http based storage back end ? if so, every image would have some url. https://some.storage.system/id.jpg which you could use directly as a download link. – Mellins

Yes, I would. But I am trying to use django-storage, and I think their API doesn't support http. @Mellins – Bridwell

Answer 1

I want to complete @Paulos Answer. You can create the proxy by using a middleware. Create a file project/middleware.py and add it to your middleware-array in the settings.py.

Then create the middleware:

import mimetypes

from storages.backends.sftpstorage import SFTPStorage
from django.http import HttpResponse



class SFTPMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
        # One-time configuration and initialization.

    def __call__(self, request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.

        SFS = SFTPStorage()
        
        response = self.get_response(request)

        path = request.get_full_path()

        if SFS.exists(path):
            file = SFS._read(path)
            type, encoding = mimetypes.guess_type(path)
            response = HttpResponse(file, content_type=type)
            response['Content-Disposition'] = u'attachment; filename="{filename}"'.format(filename=path)

        return response

EDIT: It is actually not necessary to open up a new connection on every call. You should rather create a single connection on system startup in the ready()-hook of your config.py and use this one.

Bridwell

Hi Sarius, i do have a question to your solution. I saved the middleware.py into my Django_project folder. And added the file to Middleware Array. ` "django.middleware.common.CommonMiddleware", "django_project.middleware.SFTPMiddleware", "django.middleware.csrf.CsrfViewMiddleware",` The uploaded image is still an /media/ on PC. Do I have something else to do? Thanks – Eady

Answer 2

Check django-storage setup

I feel you may have forgot to migrate your fields in django models ? In django-storage documentation on Github, you have those snippet of code.

From:

photo = models.FileField(
    storage=FileSystemStorage(location=settings.MEDIA_ROOT),
    upload_to='photos',
)

to:

photo = models.FileField(
    upload_to='photos',
)

Could that be it ? (as mention in the comment, having some snippet of code would greatly help.

SFTP access

Django-storage act has a proxy to save you files some place. I can be a s3 bucket, an http cdn like. Or in you case a SFTP server.

with other back-end supporting the HTTP protocol it is fairly easy to get the file back. As the back-end will provide you with a link directly to the content you stored.

For SFTP, this is going to be different, web pages does not natively support FTP protocol. So in order to access the file, you will have to create a proxy layer between your web pages, and the FTP server.

@action(methods=['get'], detail=True)
def download(self, request, pk=None):

    try:
        obj = ImageModel.objects.get(id=pk)
    except ImageModel.DoesNotExist:
        raise Http404

    # with some SFTP client
    # 1. check the file exist
    # 2. pull the file from the server
    # 3. attach it to the response with the proper header
    stream = sftp_client.open(obj.file.name)
    file = stream.read()

    type, encoding = mimetypes.guess_type(obj.file.name)
    response = HttpResponse(file, content_type=type)
    response['Content-Disposition'] = u'attachment; filename="{filename}'.format(
            filename=obj.file.name)
        return response
    raise Http404

Mellins

Hi Paulo, thanks. I already checked that, this doesn't solve the problem. – Bridwell

How do I create such a proxy layer? – Bridwell