Running tests against existing database using pytest-django

Question

Does anybody know how to run Django Tests using pytest-django against an existing (e.g. production) database?
I know that in general, this is not what unit tests are supposed to do, but in my case, I'm running the tests on Heroku. By default, Django creates a new test database, however, this is not possible on Heroku.

I found a solution that would work without pytest-django (python manage.py test): https://gist.github.com/gregsadetsky/5018173
but as far as I understood, pytest-django doesn't make use of the test runner defined in the Django settings.

If anybody has another approach on how to run Django tests using pytest-django on Heroku (e.g. by automating a way to create the test database) I would be happy with that solution as well.

Pola

Answer 1

Following the documentation on how to link to an existing database:

Using an existing, external database for tests

This example shows how you can connect to an existing database and use it for your tests.
This example is trivial, you just need to disable all of pytest-django and Django’s test database creation and point to the existing database. This is achieved by simply implementing a no-op django_db_setup fixture.

Put this into conftest.py:

import pytest


@pytest.fixture(scope='session')
def django_db_setup():
    settings.DATABASES['default'] = {
        'ENGINE': 'django.db.backends.mysql',
        'HOST': 'db.example.com',
        'NAME': 'external_db',
    }

Sojourn

Awesome! I don't know how I could miss that. It felt like I looked at the documentation 100 times. – Pola

This can happen :) @Pola – Sojourn

for those who are not able to make this work because of pytest.ini settings follow this github.com/pytest-dev/pytest-django/issues/643 – Yance

Where is the settings variable coming from? When I try this I'm getting a NameError because settings isn't defined. – Polyhedron

@user3250386 settings refers to the Django project's settings.py file that is used to setup the application – Sojourn

@JohnMoutafis yeah I just wasn't clear on how it's being accessed from conftest.py. I didn't know if there's an implied from django.conf import settings that wasn't being included in the example, or if there was some configuration step with pytest-django I missed that makes it automatically available to django_db_setup(), or what. (Importing settings from django.conf was making the NameError go away, but still wasn't making the example work for me...turned out my issue was related to environment variables in a .env file and a completely different solution was needed.) – Polyhedron

@JohnMoutafis actually, after some further progress, this did end up being relevant, and after sorting out my previous issue, adding from django.conf import settings did cause it to work. – Polyhedron

@Polyhedron yes that's the recommended way of importing the settings: docs.djangoproject.com/en/4.1/topics/settings/… (that's why it is no included on the example) – Sojourn

Answer 2

For those for whom the solution above does not work. You can specify a different settings.py file in pytest.ini. Like:

[pytest]
DJANGO_SETTINGS_MODULE = tests.settings

Then in the new settings.py file, import all the settings from the regular settings.py and override DATABASES:

from core.settings import *  # your usual settings

# Dadabase configuration for tests
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
        'ATOMIC_REQUESTS': True,
    }
}

In tests.settings.py you can also override other variables for testing and it will all be in one place.

I know that this won't allow you to take full advantage of the pytest fixture (such as setting the scope, and pointing to specific tests). But this is the best solution I found for specifying a custom database for tests.

External

Answer 3

Other answers should work to re-define DATABASES setting in fixture, but this seems like really poor form to me, because that should be defined in a settings file, ideally.

Scenario

You're trying to run tests with a postgres database. This particular database doesn't give your particular user permission to create new databases. This is not uncommon, as an example, using the image quay.io/sclorg/postgresql-15-c9s the user you get from using the environment variables can not create new databases. Example settings:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "foo",
        "USER": "foo",
        "PASSWORD": "secret_foo",
        "HOST": "localhost",
        "PORT": 5444
    }
}

Example error:

psycopg.errors.InsufficientPrivilege: permission denied to create database

You have a given database name (you know "foo" database exists), and you expected to use that. But still get this error, what gives?

You're getting that error because it's trying to create a test_foo database, where "foo" came from the "NAME" entry. It tries to create a new database for tests, to help you. But this is sometimes unhelpful, as in this example.

Solution

You can specify details specific to the test database nested under a "TEST" entry, as follows:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "foo",
        "USER": "foo",
        "PASSWORD": "secret_foo",
        "HOST": "localhost",
        "PORT": 5444,
        "TEST": {
            "NAME": "foo"
        }
    }
}

This will work so that you can run, for example:

py.test src/app/test/test_alan.py -k test_alan --reuse-db --migrations

The reuse-db versus create-db probably doesn't matter, but --migrations vs. --no-migrations will give different behavior only if you have not already migrated your database. The --migrations option does something equivalent to running python manage.py migrate, which is a no-op if migrations have already ran, but no-migrations will do a one-step shot to get current model state, which may still be a no-op if already migrated. But this is a warning, you might mess up your existing database with your tests if you do stuff wrong, like using --no-migrations before manually running migrations. This is why they do not make this default behavior.

This also will not work with

py.test src/app/test/test_alan.py -k test_alan -n4

Because with multiprocessing, each process needs to have its own database, which, yep, you guessed it, means it will try to create those databases. And you don't have permission to do this.

Undersea