Django test FileField using test fixtures

Question

I'm trying to build tests for some models that have a FileField. The model looks like this:

class SolutionFile(models.Model):
    '''
    A file from a solution.
    '''
    solution = models.ForeignKey(Solution)
    file = models.FileField(upload_to=make_solution_file_path)

I have encountered two problems:

1. When saving data to a fixture using ./manage.py dumpdata, the file contents are not saved, only the file name is saved into the fixture. While I find this to be the expected behavior as the file contents are not saved into the database, I'd like to somehow include this information in the fixture for tests.

2. I have a test case for uploading a file that looks like this:

   def test_post_solution_file(self):
       import tempfile
       import os
       filename = tempfile.mkstemp()[1]
       f = open(filename, 'w')
       f.write('These are the file contents')
       f.close()
       f = open(filename, 'r')
       post_data = {'file': f}
       response = self.client.post(self.solution.get_absolute_url()+'add_solution_file/', post_data,
                                   follow=True)
       f.close()
       os.remove(filename)
       self.assertTemplateUsed(response, 'tests/solution_detail.html')
       self.assertContains(response, os.path.basename(filename))
   

While this test works just fine, it leaves the uploaded file in the media directory after finishing. Of course, the deletion could be taken care of in tearDown(), but I was wondering if Django had another way of dealing with this.

One solution I was thinking of was using a different media folder for tests which must be kept synced with the test fixtures. Is there any way to specify another media directory in settings.py when tests are being run? And can I include some sort of hook to dumpdata so that it syncs the files in the media folders?

So, is there a more Pythonic or Django-specific way of dealing with unit tests involving files?

Workbench

So os.remove(foo) does't work? Does it throw an exception? Perhaps there aren't the correct privs on that directory/file to be able to delete it from inside your unit test? – Simasimah

The os.remove() part in the code deletes the file from the temp directory. In order to delete the uploaded file, I'd have to look in the media directory and follow a more complicated logic to find the exact location of the file. I'm looking for an easier, automated way to do it, if it even exists. – Workbench

D'oh, sorry! I misread your post. How about hacking settings.MEDIA_ROOT = '/path/to/project/static/and/then/alternative/storage/' and settings.MEDIA_URL = '/static/and/then/alternative/storage/' in your setUp for your tests? Hacky, but could do the job... – Simasimah

Is it possible to write a custom storage implementation and substitute it in during the tests? It looks like it should be easy enough to write an implementation that just stores uploaded files in memory. What I'm not sure about is how to substitute it in during testing. – Archdeaconry

Answer 1

Django provides a great way to write tests on FileFields without mucking about in the real filesystem - use a SimpleUploadedFile.

from django.core.files.uploadedfile import SimpleUploadedFile

my_model.file_field = SimpleUploadedFile('best_file_eva.txt', b'these are the contents of the txt file')

It's one of django's magical features-that-don't-show-up-in-the-docs :). However it is referred to here.

Hypaethral

Non-binary contents raise an error in in Python 3+; you can fix that by simply making the contents binary, like so: my_model.file_field = SimpleUploadedFile('best_file_eva.txt', b'these are the file contents!') – Jabe

Answer 2

You can override the MEDIA_ROOT setting for your tests using the @override_settings() decorator as documented:

from django.test import override_settings


@override_settings(MEDIA_ROOT='/tmp/django_test')
def test_post_solution_file(self):
  # your code here

Geffner

Answer 3

I've written unit tests for an entire gallery app before, and what worked well for me was using the python tempfile and shutil modules to create copies of the test files in temporary directories and then delete them all afterwards.

The following example is not working/complete, but should get you on the right path:

import os, shutil, tempfile

PATH_TEMP = tempfile.mkdtemp(dir=os.path.join(MY_PATH, 'temp'))

def make_objects():
    filenames = os.listdir(TEST_FILES_DIR)

    if not os.access(PATH_TEMP, os.F_OK):
        os.makedirs(PATH_TEMP)

    for filename in filenames:
        name, extension = os.path.splitext(filename)
        new = os.path.join(PATH_TEMP, filename)
        shutil.copyfile(os.path.join(TEST_FILES_DIR, filename), new)

        #Do something with the files/FileField here

def remove_objects():
    shutil.rmtree(PATH_TEMP)

I run those methods in the setUp() and tearDown() methods of my unit tests and it works great! You've got a clean copy of your files to test your filefield that are reusable and predictable.

Rootstock

I don't see how that can help me. I want to overwrite Django's media directory with a test one. And I also want to somehow export/copy the files when using ./manage.py dumpdata. – Workbench

Overwriting django's media directory is a bad idea. Unless you move the current media directory somewhere else and put it back afterward you'll never be able to run tests on your live site because it'll be a destructive operation. You could move the media folder and put it back using shutils as I suggested above. Beyond that, dumpdata will never export files for you. You'll have to write your own script or manage.py extension for that. – Rootstock

Answer 4

with pytest and pytest-django, I use this in conftest.py file:

import tempfile
import shutil
from pytest_django.lazy_django import skip_if_no_django
from pytest_django.fixtures import SettingsWrapper


@pytest.fixture(scope='session')
#@pytest.yield_fixture()
def settings():
    """A Django settings object which restores changes after the testrun"""
    skip_if_no_django()

    wrapper = SettingsWrapper()
    yield wrapper
    wrapper.finalize()


@pytest.fixture(autouse=True, scope='session')
def media_root(settings):
    tmp_dir = tempfile.mkdtemp()
    settings.MEDIA_ROOT = tmp_dir
    yield settings.MEDIA_ROOT
    shutil.rmtree(tmp_dir)


@pytest.fixture(scope='session')
def django_db_setup(media_root, django_db_setup):
    print('inject_after')

might be helpful:

1. https://dev.funkwhale.audio/funkwhale/funkwhale/blob/de777764da0c0e9fe66d0bb76317679be964588b/api/tests/conftest.py
2. https://framagit.org/ideascube/ideascube/blob/master/conftest.py
3. https://mcmap.net/q/263367/-how-to-override-a-pytest-fixture-calling-the-original-in-pytest-4

Linkoski

Answer 5

This is what I did for my test. After uploading the file it should end up in the photo property of my organization model object:

    import tempfile
    filename = tempfile.mkstemp()[1]
    f = open(filename, 'w')
    f.write('These are the file contents')
    f.close()
    f = open(filename, 'r')
    post_data = {'file': f}
    response = self.client.post("/org/%d/photo" % new_org_data["id"], post_data)
    f.close()
    self.assertEqual(response.status_code, 200)

    ## Check the file
    ## org is where the file should end up
    org = models.Organization.objects.get(pk=new_org_data["id"])
    self.assertEqual("These are the file contents", org.photo.file.read())

    ## Remove the file
    import os
    os.remove(org.photo.path)

Pyrene