"django.db.utils.ProgrammingError: relation "app_user" does not exist" during manage.py test

Question

My setup:

• Django 1.8.3
• Python 2.7.10
• Ubuntu 14.04
• django-two-factor-auth==1.2.0

I get the following error when I run python manage.py test:

Traceback (most recent call last):
  File "/src/venv/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/src/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/src/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/test.py", line 74, in execute
    super(Command, self).execute(*args, **options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/test.py", line 90, in handle
    failures = test_runner.run_tests(test_labels)
  File "/src/venv/lib/python2.7/site-packages/django/test/runner.py", line 210, in run_tests
    old_config = self.setup_databases()
  File "/src/venv/lib/python2.7/site-packages/django/test/runner.py", line 166, in setup_databases
    **kwargs
  File "/src/venv/lib/python2.7/site-packages/django/test/runner.py", line 370, in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
  File "/src/venv/lib/python2.7/site-packages/django/db/backends/base/creation.py", line 368, in create_test_db
    test_flush=not keepdb,
  File "/src/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 179, in handle
    created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 317, in sync_apps
    cursor.execute(statement)
  File "/src/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/src/venv/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/src/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 63, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "app_user" does not exist

When I drop a print(sql) statement on line 62 in django/db/backends/utils.py, I get following output:

CREATE DATABASE "test_dev"

            SELECT c.relname, c.relkind
            FROM pg_catalog.pg_class c
            LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
            WHERE c.relkind IN ('r', 'v')
                AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                AND pg_catalog.pg_table_is_visible(c.oid)
CREATE TABLE "django_migrations" ("id" serial NOT NULL PRIMARY KEY, "app" varchar(255) NOT NULL, "name" varchar(255) NOT NULL, "applied" timestamp with time zone NOT NULL)

            SELECT c.relname, c.relkind
            FROM pg_catalog.pg_class c
            LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
            WHERE c.relkind IN ('r', 'v')
                AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                AND pg_catalog.pg_table_is_visible(c.oid)
SAVEPOINT "s140275211773760_x1"
CREATE TABLE "distributedlock_lock" ("id" serial NOT NULL PRIMARY KEY, "key" varchar(255) NOT NULL, "value" varchar(255) NOT NULL, "timestamp" timestamp with time zone NULL)
RELEASE SAVEPOINT "s140275211773760_x1"
SAVEPOINT "s140275211773760_x2"
CREATE TABLE "djkombu_queue" ("id" serial NOT NULL PRIMARY KEY, "name" varchar(200) NOT NULL UNIQUE)
RELEASE SAVEPOINT "s140275211773760_x2"
SAVEPOINT "s140275211773760_x3"
CREATE TABLE "djkombu_message" ("id" serial NOT NULL PRIMARY KEY, "visible" boolean NOT NULL, "sent_at" timestamp with time zone NULL, "payload" text NOT NULL, "queue_id" integer NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x3"
SAVEPOINT "s140275211773760_x4"
CREATE TABLE "otp_static_staticdevice" ("id" serial NOT NULL PRIMARY KEY, "user_id" integer NOT NULL, "name" varchar(64) NOT NULL, "confirmed" boolean NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x4"
SAVEPOINT "s140275211773760_x5"
CREATE TABLE "otp_static_statictoken" ("id" serial NOT NULL PRIMARY KEY, "device_id" integer NOT NULL, "token" varchar(16) NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x5"
SAVEPOINT "s140275211773760_x6"
CREATE TABLE "otp_totp_totpdevice" ("id" serial NOT NULL PRIMARY KEY, "user_id" integer NOT NULL, "name" varchar(64) NOT NULL, "confirmed" boolean NOT NULL, "key" varchar(80) NOT NULL, "step" smallint NOT NULL CHECK ("step" >= 0), "t0" bigint NOT NULL, "digits" smallint NOT NULL CHECK ("digits" >= 0), "tolerance" smallint NOT NULL CHECK ("tolerance" >= 0), "drift" smallint NOT NULL, "last_t" bigint NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x6"
CREATE INDEX "djkombu_queue_name_1c24e49fd475ad53_like" ON "djkombu_queue" ("name" varchar_pattern_ops)
ALTER TABLE "djkombu_message" ADD CONSTRAINT "djkombu_message_queue_id_12778caea7843dd_fk_djkombu_queue_id" FOREIGN KEY ("queue_id") REFERENCES "djkombu_queue" ("id") DEFERRABLE INITIALLY DEFERRED
CREATE INDEX "djkombu_message_46cf0e59" ON "djkombu_message" ("visible")
CREATE INDEX "djkombu_message_df2f2974" ON "djkombu_message" ("sent_at")
CREATE INDEX "djkombu_message_75249aa1" ON "djkombu_message" ("queue_id")
ALTER TABLE "otp_static_staticdevice" ADD CONSTRAINT "otp_static_staticdevice_user_id_39a61f1bd3ec970d_fk_app_user_id" FOREIGN KEY ("user_id") REFERENCES "ff_user" ("id") DEFERRABLE INITIALLY DEFERRED

So it is clear to me that my tests blow up while the test database is being setup. Specifically, the attempt to create a foreign key constraint between the otp_static_staticdevice table and my app's app_user table fails.

My immediate question is, why does django create the OTP table before my app's table? My assumption is that the OTP app is listed first in my INSTALLED_APPS. But this is not the case:

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.admin',
    'django.contrib.humanize',
    'app',
    ...
    'django_otp',
    'django_otp.plugins.otp_static',
    'django_otp.plugins.otp_totp',
    'two_factor',
    ...
]

Next, I look at django/core/management/commands/migrate.py, trying to find out how django determines its order for migrating apps.

Plopping a pdb.set_trace() statement on line 264 (https://github.com/django/django/blob/1.8.3/django/core/management/commands/migrate.py#L264) and looking to see what app_labels contains, I get:

set(['djangosaml2', 'django_ace', 'recurly', 'staticfiles', 'distributedlock', 'app_overrides', 'messages', 'django_otp', 'kombu_transport_django', 'otp_totp', 'compressor', 'otp_static', 'humanize', 'ajax_select', 'django_extensions', 'import_export', 'raven_compat', 'crispy_forms', 'emoji'])

This is as far as I have gotten before I decided to ask for help. Does anyone know how Django might end up not creating all the project's apps in the correct order so that decency conflicts do not occur?

Bolognese

try ./manage.py makemigrations app first, then continue – Batty

@Bolognese Did you solve this problem? I'm facing the same thing, except it complains about a different relation. – Letaletch

Answer 1

Got the same issue, and since it happens on ./manage.py test, your migrations may be broken.
Since Django 1.7, there is a new setting called MIGRATION_MODULES, in which you configure your app's migration modules.
Adding the following workaround in settings.py (found here) skips migrations on tests, and solved it for me:

class DisableMigrations(object):

    def __contains__(self, item):
        return True

    def __getitem__(self, item):
        return "notmigrations"

MIGRATION_MODULES = DisableMigrations()

Mankind

After adding these lines in settings.py file it returns following error- ModuleNotFoundError: No module named 'notmigrations' – Sackey

Answer 2

This should be solved by running:

python manage.py migrate auth
python manage.py migrate app
python manage.py migrate

Deidradeidre

Answer 3

python manage.py migrate auth

This works.

Mymya

Answer 4

You have not given any code so its difficult to understand your models relations. But can I guess that you have passed the ForeignKey model as an instance? Try to give it as a string

class MyModel(models.Model):
    fk_field = models.ForeignKey('path.to.other.model')    # same as from path.models import model

This usually works out. Hope it does for you too!

Fleming

Answer 5

I had a similar issue with manage.py test when I ran it in gitlab CI jobs, I was able to fix this by creating a variable called CI in my base.py settings file

base.py

"""
Set CI Variable to True when running: "manage.py test"
"""
CI = True  

and in my settings.py:

if CI:
    class DisableMigrations(object):

        def __contains__(self, item):
            return True

        def __getitem__(self, item):
            return None # For Django 1.10+

MIGRATION_MODULES = DisableMigrations()

Castellano

Answer 6

Ensure all your app directories have migrations/ folder and __init__.py file in it. If not, make a migrations/ folder and create __init__.py file inside. If you using git, add !migrations/__init__.py to the .gitignore file

Metallist

Tried everything else. This is what worked for me. Thanks. – Kroon

Answer 7

I've got same error beacuse one of my django applications have not migrations directory. Try to watch carefully all your apps folders which are present in INSTALLED_APPS.

Lewendal

Answer 8

In my case, migrations didn't run because I used unittest.TestCase instead of django.test.TestCase.

Solution: replace

from unittest import TestCase

with

from django.test import TestCase

Roodepoortmaraisburg