Problems with contenttypes when loading a fixture in Django

Question

I am having trouble loading Django fixtures into my MySQL database because of contenttypes conflicts. First I tried dumping the data from only my app like this:

./manage.py dumpdata escola > fixture.json

but I kept getting missing foreign key problems, because my app "escola" uses tables from other applications. I kept adding additional apps until I got to this:

./manage.py dumpdata contenttypes auth escola > fixture.json

Now the problem is the following constraint violation when I try to load the data as a test fixture:

IntegrityError: (1062, "Duplicate entry 'escola-t23aluno' for key 2")

It seems the problem is that Django is trying to dynamically recreate contenttypes with different primary key values that conflict with the primary key values from the fixture. This appears to be the same as bug documented here: http://code.djangoproject.com/ticket/7052

The problem is that the recommended workaround is to dump the contenttypes app which I'm already doing!? What gives? If it makes any difference I do have some custom model permissions as documented here: http://docs.djangoproject.com/en/dev/ref/models/options/#permissions

Narthex

Answer 1

manage.py dumpdata --natural will use a more durable representation of foreign keys. In django they are called "natural keys". For example:

• Permission.codename is used in favour of Permission.id
• User.username is used in favour of User.id

Read more: natural keys section in "serializing django objects"

Some other useful arguments for dumpdata:

• --indent=4 make it human readable.
• -e sessions exclude session data
• -e admin exclude history of admin actions on admin site
• -e contenttypes -e auth.Permission exclude objects which are recreated automatically from schema every time during syncdb. Only use it together with --natural or else you might end up with badly aligned id numbers.

Samford

I am using this to get natural keys for a content_type attribute, but I get this error when trying to load back the fixtures. TypeError: string indices must be integers, not str Any idea why ? – Narghile

@skyjur Why always use -e contenttypes -e auth.permission with --natural? I just tried without the --natural option and it worked. Also the documentation here says one should use this option if DUMPING auth.permission and contenttypes. – Timaru

@winirvana because after you start from scratch and do syncdb, newly created ContentType and Permission are not guaranteed to get same id as they had before. Your data dump contains ids which might reference different objects on anather database where you will loaddata. It might worked for you because of one of these reasons: 1) your data didn't had any reference to these objects 2) original ids of Permission/ContentTypes were preserved 3) your loaddata was succesful but you actually have corrupt data due to objects referring to wrong objects and you don't yet know about it – Samford

Flag --natural is now deprecated in favor of --natural-foreign (and --natural-primary) – Clarabelle

Final command could be: manage.py dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.Permission --indent 4 > project_dump.json – Carlyncarlynn

--natural has now been completely removed, not just deprecated. Use --natural-foreign or --natural-primary instead. – Prouty

Answer 2

The answers here all old... As of 2017, the best answer is:

manage.py dumpdata --natural-foreign --natural-primary -e contenttypes -e auth.Permission --indent 4

Colligan

Answer 3

Yes, this is really irritating. For a while I worked around it by doing a "manage.py reset" on the contenttypes app prior to loading the fixture (to get rid of the automatically-generated contenttypes data that differed from the dumped version). That worked, but eventually I got sick of the hassles and abandoned fixtures entirely in favor of straight SQL dumps (of course, then you lose DB portability).

update - the best answer is to use the --natural flag to dumpdata, as noted in an answer below. That flag did not exist yet when I wrote this answer.

Riot

I was running into this too, resetting the contenttypes app worked for me as well. Thanks for the tip! – Identical

How did you reset them? In test case class? Give me an example please – Bouleversement

I don't use fixtures for unittests, I generally create test data using the ORM in a setup() method because it's easier to keep in sync with the tests. So I never had to do this in a TestCase class, though I'm sure if you poke around in the code for Django's TestCase class you could figure out how to make a reset happen post syncdb and prior to fixture loading in a subclass. For me it was just "./manage.py reset contenttypes" in a bash script prior to "./manage.py loaddata my_fixture". – Riot

Answer 4

Try skipping contenttypes when creating fixture:

./manage.py dumpdata --exclude contenttypes > fixture.json

It worked for me in a similar situation for unit tests, your insight regarding the contenttypes really helped!

Beshrew

Answer 5

I was not using MySQL but instead importing some data from a live server into sqlite. Clearing the contenttypes app data before performing loaddata did the trick:

from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()
quit()

And then

python manage.py loaddata data.json

Basuto

django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. – Spermatozoon

It would probably work best within the handle of a custom management command. – Spermatozoon

Answer 6

I have resolved this issue in my test cases by resetting the contenttypes app from the unit test prior to loading my dump file. Carl suggested this already using the manage.py command and I do the same thing only using the call_command method:

>>> from django.core import management
>>> management.call_command("flush", verbosity=0, interactive=False)
>>> management.call_command("reset", "contenttypes", verbosity=0, interactive=False)
>>> management.call_command("loaddata", "full_test_data.json", verbosity=0)

My full_test_data.json fixture contains the contenttypes app dump that corresponds to the rest of the test data. By resetting the app before loading, it prevents the duplicate key IntegrityError.

Kilohertz

Answer 7

You need to use natural keys to represent any foreign key and many-to-many relationships. Moreover, it might be a good idea to exclude the session table in the sessions app, and the logentry table in the admin app.

Django 1.7+

python manage.py dumpdata --natural-foreign --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json

Django <1.7

python manage.py dumpdata --natural --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json

According to the Django documentation, --natural has been deprecated in version 1.7, so the option --natural-foreign should be used instead.

You can also omit the primary key in the serialized data of this object since it can be calculated during deserialization by passing the --natural-primary flag.

python manage.py dumpdata --natural-foreign --natural-primary --exclude contenttypes --exclude auth.permission --exclude admin.logentry --exclude sessions.session --indent 4 > fixture.json

Gouveia

Answer 8

python manage.py dumpdata --natural-primary --exclude=contenttypes --exclude=auth.Permission --exclude=admin.logentry --exclude=sessions.session --indent 4 > initial_data.json

This works for me. Here I am excluding everything bubt the actual models.

• If you see any other model other than the models that you created you can safely exclude those. One drawback of this approach is you loose on log data as well as auth data.

Ancilin

Answer 9

./manage.py dumpdata app.Model --natural-foreign

will change

  "content_type": 123

to

  "content_type": [
    "app_label",
    "model"
  ],

And fixture works for TestCase now

Daladier

Answer 10

Django 2.2.5

python manage.py dumpdata --exclude=contenttypes > datadump.json

it helped me

Majestic

It will raise problem when loaddata, maybe mismatch with contenttype in new database – Wear

Answer 11

It's really, really annoying .. I get bitten by this every single time.

I tried to dumpdata with --exclude contenttypes and --natural, I always get problems..

What works best for me is simply doing a truncate table django_content_type; after the syncdb and THEN load the data.

Of course for initial_data.json autoloading you're fallball.

Greiner

For me, truncating the table prior to loaddata just causes different errors. No luck with this technique. – Elnora

Answer 12

I tried every method from above, Nothing worked for me. I have to exclude the complete auth model and works fine.

python manage.py dumpdata --natural-primary --exclude=contenttypes --exclude=auth --exclude=admin.logentry --exclude=sessions.session --indent 4 > live.json

Arawak

Answer 13

I'm going to give another possible answer that I just figured out. Maybe it'll help the OP, maybe it'll help somebody else.

I've got a many-to-many relationship table. It has a primary key and the two foreign keys to the other tables. I found that if I have an entry in the fixture whose two foreign keys are the same as another entry already in the table with a different pk, it will fail. M2M relationship tables have a "unique together" for the two foreign keys.

So, if it's a M2M relationship that is breaking, look at the foreign keys it's adding, look at your database to see if that pair of FKs are already listed under a different PK.

Must

Answer 14

I had encountered similar error sometimes ago. It turned out that I was trying to load the fixtures before creating the necessary tables. So I did:

$ python manage.py makemigrations
$ python manage.py migrate
$ python manage.py loaddata fixtures/initial_data.json

And it worked like a charm

Dafodil

Answer 15

In my case I had dumped the data from auth (./manage.py dumpddata auth > fixtures/auth.json) to use the fixture for testing purposes.

The development continued and I removed most of the models I had defined in models.py and this is when I started to see this annoying problem.

My solution was regenerating the auth.json fixture again. This one had removed lots of entries in auth.permission related to the old models I had.

Tar

Answer 16

I've fixed this by adding in my tests setUp and tearDown

from django.core import management

=====

def setUp(self):
    management.call_command("loaddata", "all-data.yaml", verbosity=0)
    super(login_page_test, self).setUp()

def tearDown(self):
    management.call_command("flush", verbosity=0, interactive=False)
    super(login_page_test, self).setUp()

Miguelmiguela

Answer 17

I used pgloader, just take a few seconds to migrate successfully:

$ pgloader project.load

project.load file with:

load database
     from sqlite:////path/to/dev.db
     into postgresql://user:pwd@localhost/db_name

 with include drop, create tables, create indexes, reset sequences

   set work_mem to '16MB', maintenance_work_mem to '512 MB';

Mononuclear

Answer 18

You can try to exclude some types of data in the database when dumping data.

First, dump data:

python manage.py dumpdata --indent 4 > datadump.json

A few lines from this file for example:

[

{ "model": "admin.logentry", "pk": 1, "fields": { "action_time": "2023-09-28T08:33:24.827Z", "user": 1, "content_type": 3, "object_id": "1", "object_repr": "user", "action_flag": 1, "change_message": "[{"added": {}}]" } },

...

{ "model": "auth.permission", "pk": 1, "fields": { "name": "Can add log entry", "content_type": 1, "codename": "add_logentry" } },

...

{ "model": "auth.group", "pk": 1, "fields": { "name": "user", "permissions": [ 25, 26, 27 ] } },

...

]

Try to search and see the text like this "model" in the json dump file.

You can also find all types of content in the contenttype model, a table for the this model is django_content_type:

SELECT * FROM django_content_type ORDER BY id;

An example of data from this table:

+----+--------------+----------------+
| id | app_label    | model          |
+----+--------------+----------------+
|  1 | admin        | logentry       |
|  2 | auth         | permission     |
|  3 | auth         | group          |
|  4 | contenttypes | contenttype    |
|  5 | sessions     | session        |
|  6 | user         | user           |
|  7 | user         | historicaluser |
|  8 | constance    | constance      |
|  9 | tracking     | pageview       |
| 10 | tracking     | visitor        |
+----+--------------+----------------+

After you can exclude this type of data by adding several --exclude to the manage.py dumpdata

For example:

• "model": "auth.permission", can be excluded like this: --exclude auth.permission. Using this option --exclude auth you can exclude all models for application auth, if there are such models, in this case these will be the permission and the group models.
• "model": "contenttypes.contenttype", can be excluded like this: --exclude contenttypes.contenttype.
• "model": "user.historicaluser", can be excluded like this: --exclude user.historicaluser.
• "model": "sessions.session", can be excluded like this: --exclude sessions.session or --exclude sessions.
• "model": "tracking.pageview", can be excluded like this: --exclude tracking.pageview. If you use --exclude tracking, both models tracking.pageview and tracking.visitor will be excluded.

The final code for 5 lines above:

python manage.py dumpdata --exclude auth.permission --exclude contenttypes --exclude user.historicaluser --exclude sessions.session --exclude tracking --indent 4 > dumpdata.json

You can also combine the dump script with --natural-foreign and --natural-primary options, read more here: https://docs.djangoproject.com/en/5.0/topics/serialization/#natural-keys.

Lyford