django-storages and amazon s3 - suspiciousoperation

Question

I'm using django-storages with Amazon S3. I see the following error somewhat intermittently:

name = self._normalize_name(self._clean_name(name))\n\n  File \"/app/.heroku/venv/lib/python2.7/site-packages/storages/backends/s3boto.py\", line 237, in _normalize_name\n    name)\n\nSuspiciousOperation: Attempted access to 'https:/plantvillage.s3.amazonaws.com/avatar/hans9_avatar.jpg'

Note the single / after https:.

Does anyone know why this shows up? It doesn't happen all the time. I can successfully do this in other cases.

Ledbetter

Answer 1

_normalize_name does a lot of fancy and mostly unnecessary on Django stuff with the URL. In my case I just override the S3BotoStorage like this:

class S3CustomStorage(S3BotoStorage):
def _normalize_name(self, name):
    """
    Get rid of this crap: https://mcmap.net/q/1215630/-django-storages-and-amazon-s3-suspiciousoperation
    """
    return name

Then use it in the storage property:

ImageField(storage=S3CustomStorage())

And it worked for django simple ImageField with this base configuration:

AWS_ACCESS_KEY_ID = 'TTTT'
AWS_SECRET_ACCESS_KEY = 'XXXX'
AWS_STORAGE_BUCKET_NAME = 'ZZZZ'

Klump

Answer 2

When you use default_storage methods make sure to use the file.name:

Correct:

default_storage.delete(file.name)

Wrong:

default_storage.delete(file.url)

Wrong:

default_storage.delete(file)

All three examples above work with local files, but when using s3 you will run into this error unless you use file.name.

Dreadfully

file.url is processed differently than file.name. I've also done this, but it seems more like a hack than an actual solution. Are there any sources which indicate this is indeed correct, besides the fact that this method does not raise an exception? – Aerate

Not that i am aware of. – Dreadfully

Answer 3

I haven't gotten S3 storage working on my own project yet, but I did just run across this error, and might be able to point you in the right direction.

If you look at S3BotoStorage._clean_name(), it's just: return os.path.normpath(name).replace('\\', '/'). os.path.normpath() converts the // in your URL to \\, and then .replace() converts that to \. Then, S3BotoStorage._normalize_name() checks to make sure this broken URL is part of the location it represents, which of course it's not. That's where the SuspiciousOperation error is being raised.

So 'name' looks like it's meant to be a local path, instead of the entire AWS URL. In my case, the immediate cause was FILEBROWSER_DIRECTORY = MEDIA_URL + "uploads/" in settings.py, which I had tried hoping to fix a different error about a missing upload folder.

Airla

Answer 4

Setting

MEDIA_ROOT=''

fixed the problem for me.

Krueger

Answer 5

I fixed this, adding SuspiciousOperation on except:

class S3CustomStorage(S3BotoStorage):
    def _normalize_name(self, name):
        try:
            return safe_join(self.location, name)
        except (SuspiciousOperation, ValueError):
            return ""

Flagellate