Django manage.py test cannot load fixture properly

Question

I've written Django tests using django.test.TestCase, and I'd like to use a fixture with all of my current database data to run the tests. However, if I create the fixture as follows:

python manage.py dumpdata --indent=3 > myapp/fixtures/test_data.json

when I then run the tests using python manage.py test myapp, I get the following error:

Problem installing fixture...(traceback)
IntegrityError: Could not load auth.Permission(pk=42): duplicate key value violates unique constraint "auth_permission_content_type_id_codename_key"
DETAIL:  Key (content_type_id, codename)=(14, add_record) already exists.

I read somewhere on SO that this might be caused by a pk conflict so I then tried re-creating the fixture with:

python manage.py dumpdata --natural --indent=3 > myapp/fixtures/test_data.json

But now running the test gives me:

Problem installing fixture...(traceback)
DeserializationError: 'NoneType' object has no attribute '_meta'

I've also tried variously excluding (using the --exclude option) auth.permission and contenttypes (or both simultaneously), but then I get complaints about missing permissions (Key (permission_id)=(44) is not present in table "auth_permission".) or missing content types (DeserializationError: ContentType matching query does not exist.)

The thing is, I need permissions anyway because my tests are partly to verify that only users with specific permissions can access certain views.

I don't understand why this is happening, to be honest--my impression was that the test runner starts with a completely clean database and loads EVERYTHING from my fixture, but reading posts like this one: Django unit-testing with loading fixtures for several dependent applications problems makes it seem like maybe that's not the case.

How can I get around this? I'd much rather not have to write stuff like User.objects.create_user(.. tons of times under def setUp(self): in my tests just to have enough objects for them to run properly...

Procarp

Answer 1

The issue still seems to have been present, even with --natural. However, it seems to be resolved in django1.9 with new flags: dumpdata --natural-foreign --natural-primary

see https://code.djangoproject.com/ticket/21278#comment:5

Secunda

For anyone finding this with a similar problem, here's a KEY insight: if you models contain an M2M relation, do NOT dump the M2M table into your test fixture! Otherwise, the M2M will first be populate when the related objects are created, and then AGAIN when the M2M data is loaded from the fixture, thus causing an integrity error. – Secunda

Answer 2

When you run a test the initial_data fixtures will be loaded (by syncdb).

For me dumpdata works with the --natural argument, excluding the contenttypes and then deleting some auth.permission entries manually that only these remain:

{
    "pk": 1, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_permission", 
        "name": "Can add permission", 
        "content_type": [
            "auth", 
            "permission"
        ]
    }
},
{
    "pk": 2, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_permission", 
        "name": "Can change permission", 
        "content_type": [
            "auth", 
            "permission"
        ]
    }
},
{
    "pk": 3, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_permission", 
        "name": "Can delete permission", 
        "content_type": [
            "auth", 
            "permission"
        ]
    }
},
{
    "pk": 4, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_group", 
        "name": "Can add group", 
        "content_type": [
            "auth", 
            "group"
        ]
    }
},
{
    "pk": 5, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_group", 
        "name": "Can change group", 
        "content_type": [
            "auth", 
            "group"
        ]
    }
},
{
    "pk": 6, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_group", 
        "name": "Can delete group", 
        "content_type": [
            "auth", 
            "group"
        ]
    }
},
{
    "pk": 7, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_user", 
        "name": "Can add user", 
        "content_type": [
            "auth", 
            "user"
        ]
    }
},
{
    "pk": 8, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_user", 
        "name": "Can change user", 
        "content_type": [
            "auth", 
            "user"
        ]
    }
},
{
    "pk": 9, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_user", 
        "name": "Can delete user", 
        "content_type": [
            "auth", 
            "user"
        ]
    }
},
{
    "pk": 10, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_contenttype", 
        "name": "Can add content type", 
        "content_type": [
            "contenttypes", 
            "contenttype"
        ]
    }
},
{
    "pk": 11, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_contenttype", 
        "name": "Can change content type", 
        "content_type": [
            "contenttypes", 
            "contenttype"
        ]
    }
},
{
    "pk": 12, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_contenttype", 
        "name": "Can delete content type", 
        "content_type": [
            "contenttypes", 
            "contenttype"
        ]
    }
},
{
    "pk": 13, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_session", 
        "name": "Can add session", 
        "content_type": [
            "sessions", 
            "session"
        ]
    }
},
{
    "pk": 14, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_session", 
        "name": "Can change session", 
        "content_type": [
            "sessions", 
            "session"
        ]
    }
},
{
    "pk": 15, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_session", 
        "name": "Can delete session", 
        "content_type": [
            "sessions", 
            "session"
        ]
    }
},
{
    "pk": 16, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_site", 
        "name": "Can add site", 
        "content_type": [
            "sites", 
            "site"
        ]
    }
},
{
    "pk": 17, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_site", 
        "name": "Can change site", 
        "content_type": [
            "sites", 
            "site"
        ]
    }
},
{
    "pk": 18, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_site", 
        "name": "Can delete site", 
        "content_type": [
            "sites", 
            "site"
        ]
    }
},

I don't understand excactly why, but it works. I'd try to compare my fixture with a dump of a fresh database directly after syncdb and decide then what to delete or edit in my fixture. Hope this helps in your case.

Pirogue