Django Unit Testing taking a very long time to create test database

Question

For some time now, my unit testing has been taking a longer than expected time. I have tried to debug it a couple of times without much success, as the delays are before my tests even begin to run. This has affected my ability to do anything remotely close to test driven development (maybe my expectations are too high), so I want to see if I can fix this once and for all.

When a run a test, there is a 70 to 80sec delay between the start and the actual beginning of the test. For example, if I run a test for a small module (using time python manage.py test myapp), I get

<... bunch of unimportant print messages I print from my settings>

Creating test database for alias 'default'...
......
----------------------------------------------------------------
Ran 6 tests in 2.161s

OK
Destroying test database for alias 'default'...

real    1m21.612s
user    1m17.170s
sys     0m1.400s

About 1m18 of the 1m:21 are between the

Creating test database for alias 'default'...

and the

.......

line. In other words, the test takes under 3sec, but the database initialization seems to be taking 1:18min

I have about 30 apps, most with 1 to 3 database models so this should give an idea of the project size. I use SQLite for unit testing, and have implemented some of the suggested improvements. I cannot post my whole setting file, but happy to add any information that is required.

I do use a runner

from django.test.runner import DiscoverRunner
from django.conf import settings

class ExcludeAppsTestSuiteRunner(DiscoverRunner):
    """Override the default django 'test' command, exclude from testing
    apps which we know will fail."""

    def run_tests(self, test_labels, extra_tests=None, **kwargs):
        if not test_labels:
            # No appnames specified on the command line, so we run all
            # tests, but remove those which we know are troublesome.
            test_labels = (
                'app1',
                'app2',
                ....
                )
            print ('Testing: ' + str(test_labels))

        return super(ExcludeAppsTestSuiteRunner, self).run_tests(
                test_labels, extra_tests, **kwargs)

and in my settings:

TEST_RUNNER = 'config.test_runner.ExcludeAppsTestSuiteRunner'

I have also tried using django-nose with django-nose-exclude

I have read a lot about how to speed up the test themselves, but have not found any leads on how to optimize or avoid the database initialization. I have seen the suggestions on trying not to test with the database but I cannot or don't know how to avoid that completely.

Please let me know if

1. This is normal and expected
2. Not expected (and hopefully a fix or lead on what to do)

Again, I don't need help on how to speed up the test themselves, but the initialization (or overhead). I want the example above to take 10sec instead of 80sec.

Many thanks

I run the test (for single app) with --verbose 3 and discovered this is all related to migrations:

  Rendering model states... DONE (40.500s)
  Applying authentication.0001_initial... OK (0.005s)
  Applying account.0001_initial... OK (0.022s)
  Applying account.0002_email_max_length... OK (0.016s)
  Applying contenttypes.0001_initial... OK (0.024s)
  Applying contenttypes.0002_remove_content_type_name... OK (0.048s)
  Applying s3video.0001_initial... OK (0.021s)
  Applying s3picture.0001_initial... OK (0.052s)
  ... Many more like this

I squashed all my migrations but still slow.

Schoolbook

Very helpful. Especially the verbose option. On our project there are a couple of hundreds of migrations, some of which apparently take up to a second to complete. For those using PyCharm (Pro), you can add --verbose 3 (or -v 3) to the run-configuration for your test (under "Options:"). – Inherit

Answer 1

The final solution that fixes my problem is to force Django to disable migration during testing, which can be done from the settings like this

TESTING = 'test' in sys.argv[1:]
if TESTING:
    print('=========================')
    print('In TEST Mode - Disableling Migrations')
    print('=========================')

    class DisableMigrations(object):

        def __contains__(self, item):
            return True

        def __getitem__(self, item):
            return None

    MIGRATION_MODULES = DisableMigrations()

or use https://pypi.python.org/pypi/django-test-without-migrations

My whole test now takes about 1 minute and a small app takes 5 seconds.

In my case, migrations are not needed for testing as I update tests as I migrate, and don't use migrations to add data. This won't work for everybody

Schoolbook

Awesome, you must have a lot of migrations! :-) – Dardanus

Not really as I have squashed all of mine. But I do have an issue ( for another post ) where Django-allauth keeps creating a new migration every time I makemigrations . But there is a lot of posts on Django's 1.8 slow migrations. I believe 1.9 fixed some of it – Schoolbook

Yes indeed, I read about this somewhere. To overcome this issue I just use pytest-django which has an option --nomigrations that creates the database from the models directly. Apparently that was the default behavior of Django until 1.6. I know it's useful to run the migrations once before deploying but not for every test run! – Dardanus

wouldn't --keepdb do the same? – Pustule

This was some time ago so I don't remember the specifics but what I remember is that even doing a single test had the extra delay so keeping the db, as I understand it, wouldn't help with that. Regardless, I don't want to deal with some out of sync database so the solution below has worked great for me. – Schoolbook

That is correct, --keepdb doesn't help resolve the slow migration issue. This solution does. Thanks! – Baalman

With django 2.1 I had to change return "notmigrations" to return None, otherwise it complained that ModuleNotFoundError: No module named 'notmigrations' – Dolphin

Answer 2

Summary

Use pytest !

Operations

1. pip install pytest-django
2. pytest --nomigrations instead of ./manage.py test

Result

• ./manage.py test costs 2 min 11.86 sec
• pytest --nomigrations costs 2.18 sec

Hints

• You can create a file called pytest.ini in your project root directory, and specify default command line options and/or Django settings there.

  # content of pytest.ini
  [pytest]
  addopts = --nomigrations
  DJANGO_SETTINGS_MODULE = yourproject.settings
  

  Now you can simply run tests with pytest and save you a bit of typing.

• You can speed up the subsequent tests even further by adding --reuse-db to the default command line options.

  [pytest]
  addopts = --nomigrations --reuse-db
  

  However, as soon as your database model is changed, you must run pytest --create-db once to force re-creation of the test database.

• If you need to enable gevent monkey patching during testing, you can create a file called pytest in your project root directory with the following content, cast the execution bit to it (chmod +x pytest) and run ./pytest for testing instead of pytest:

  #!/usr/bin/env python
  # -*- coding: utf-8 -*-
  # content of pytest
  from gevent import monkey
  
  monkey.patch_all()
  
  import os
  
  os.environ.setdefault("DJANGO_SETTINGS_MODULE", "yourproject.settings")
  
  from django.db import connection
  
  connection.allow_thread_sharing = True
  
  import re
  import sys
  
  from pytest import main
  
  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(main())
  

  You can create a test_gevent.py file for testing whether gevent monkey patching is successful:

  # -*- coding: utf-8 -*-
  # content of test_gevent.py
  import time
  from django.test import TestCase
  from django.db import connection
  import gevent
  
  
  def f(n):
      cur = connection.cursor()
      cur.execute("SELECT SLEEP(%s)", (n,))
      cur.execute("SELECT %s", (n,))
      cur.fetchall()
      connection.close()
  
  
  class GeventTestCase(TestCase):
      longMessage = True
  
      def test_gevent_spawn(self):
          timer = time.time()
          d1, d2, d3 = 1, 2, 3
          t1 = gevent.spawn(f, d1)
          t2 = gevent.spawn(f, d2)
          t3 = gevent.spawn(f, d3)
          gevent.joinall([t1, t2, t3])
          cost = time.time() - timer
          self.assertAlmostEqual(cost, max(d1, d2, d3), delta=1.0,
                                 msg='gevent spawn not working as expected')
  

References

• pytest-django documentation
• pytest documentation

Bandit

Also, if you're tired of looking at little grey dots all day, pip install pytest-sugar and suddenly testing is pretty! – Cari

I tried this approach but I got an error regarding db permissions. I think I have to wrap all my test cases with a decorator for this to work. pytest-django.readthedocs.io/en/latest/…. But this seems too intrusive. – Celia

Answer 3

use ./manage.py test --keepdb when there are no changes in the migration files

Profitable

Are there any potential pitfalls to this approach? – Perceive

My only concern is that: would the database be cleared after every test case run? but otherwise, you save a lot of time running DDL SQL queries from each migration. It is particularly slow when you don't have a SSD. It should also speed up if you have say 100's of migrations. This is a temporary fix for running tests one after the other quickly during development without having to wait for 5 to 10 minutes between each. – Profitable

Answer 4

Database initialization indeed takes too long...

I have a project with about the same number of models/tables (about 77), and approximately 350 tests and takes 1 minute total to run everything. Deving in a vagrant machine with 2 cpus allocated and 2GB of ram. Also I use py.test with pytest-xdist plugin for running multiple tests in parallel.

Another thing you can do is tell django reuse the test database and only re-create it when you have schema changes. Also you can use SQLite so that the tests will use an in-memory database. Both approaches explained here: https://docs.djangoproject.com/en/dev/topics/testing/overview/#the-test-database

EDIT: In case none of the options above work, one more option is to have your unit tests inherit from django SimpleTestCase or use a custom test runner that doesn't create a database as explained in this answer here: django unit tests without a db.

Then you can just mock django calls to the database using a library like this one (which admittingly I wrote): https://github.com/stphivos/django-mock-queries

This way you can run your unit tests locally fast and let your CI server worry about running integration tests that require a database, before merging your code to some stable dev/master branch that isn't the production one.

Dardanus

Thanks for the answer. I just corrected the OP to indicate I am already using SQLite in-memory. I just tried the --keepdb but does not seem to have any effect, maybe because I am using SQLite in-memory. Is there a way to use SQLite but with a database file, so that it can be preserved? My overall test (174 tests) runs in 1:18 initialization plus 1min = 2:18min so that is acceptable. In your case, your initialization is clearly a lot smaller. I don't think the database initialization can be run in parallel, so it seems like you have zero overhead. Is that the case? – Schoolbook

I would say in general, database initialization would be more I/O intensive and running the tests more memory and cpu intensive - especially for distributed testing. But you are using SQLite in-memory db so not sure how much I/O is involved for db initialization. Never had so many apps in one project, not sure how much overhead there is for model discovery in each of them. Your specs? Also do you create a large number of test data before running the tests? – Dardanus

We seem to have abut the same number of models, give or take, and you have more tests than me, so I hope to be able to get your results. I don't know what you mean by "Spec" but no, I don't have any fixtures. All data is created as part of the test themselves, or during setUp() – Schoolbook

Sorry I meant to ask you if your dev machine has limited specs. Updated my answer with a solution for that. – Dardanus

No, nothing I have tried, including squashing all migrations have tried. I need to try the mocking library but haven't been able to do it – Schoolbook

Added some notes un running test with --verbose 3 which shows that this is all due to migration times when creating the database – Schoolbook

Answer 5

If you're using Postgres, use a Postgres template to store a copy of an otherwise empty database with all migrations applied. Then configure Django to use that template during test database creation.

One way to do this:

1. Run Django's test command with the --keepdb argument.
2. On Postgres, rename the created test database using the query

alter database "test_your_db_name" rename to "test_your_db_name_template";

3. On Postgres, make the database a template using the query

alter database "test_your_db_name_template" IS_TEMPLATE = true

4. Tweak your test settings using something like:

DATABASES["default"]["TEST"] = {}
DATABASES["default"]["TEST"]["TEMPLATE"] = "test_your_db_name_template"

5. Run the Django tests again without the --keepdb argument. Since the migrations were already applied to the template database, they will not have to run again, so the tests will start pretty much immediately.

See also the documentation:

• Django TEMPLATE database setting

Ornelas

Answer 6

I also run into issue, One solution what I did is to subclass the Django.TestCase --> create subclass of Django.TestCase

and overwritten the method like :

@classmethod def _databases_support_transactions(cls): return True

the backend DB is apache cassandra ..

Tengdin