In Django, getting a "Error: Unable to serialize database" when trying to dump data?

Question

I'm getting an error when I'm trying to dump data to a JSON fixture in Djanog 1.2.1 on my live server. On the live server it's running MySQL Server version 5.0.77 and I imported a lot of data to my tables using the phpMyAdmin interface. The website works fine and Django admin responds as normal. But when I try and actually dump the data of the application that corresponds to the tables I get this error:

$ python manage.py dumpdata --indent=2 gigs > fixtures/gigs_100914.json 
/usr/local/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
  from sets import ImmutableSet
Error: Unable to serialize database: Location matching query does not exist.

My Django model for 'gigs' that I'm trying to dump from looks like this in the models.py file:

from datetime import datetime
from django.db import models

class Location(models.Model):
    name = models.CharField(max_length=120, blank=True, null=True)

    class Meta:
        ordering = ['name']

    def __unicode__(self):
        return "%s (%s)" % (self.name, self.pk)

class Venue(models.Model):
    name = models.CharField(max_length=120, blank=True, null=True)
    contact = models.CharField(max_length=250, blank=True, null=True)
    url = models.URLField(max_length=60, verify_exists=False, blank=True, null=True) # because of single thread problems, I left this off (http://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.URLField.verify_exists)

    class Meta:
        ordering = ['name']

    def __unicode__(self):
        return "%s (%s)" % (self.name, self.pk)

class Gig(models.Model):
    date = models.DateField(blank=True, null=True)
    details = models.CharField(max_length=250, blank=True, null=True)
    location = models.ForeignKey(Location)
    venue = models.ForeignKey(Venue)

    class Meta:
        get_latest_by = 'date'
        ordering = ['-date']

    def __unicode__(self):
        return u"%s on %s at %s" % (self.location.name, self.date, self.venue.name)

Like I say, Django is fine with the data. The site works fine and the relationships seem to operate absolutely fine. When a run the command to get what SQL Django is using:

$ python manage.py sql gigs
/usr/local/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
  from sets import ImmutableSet
BEGIN;CREATE TABLE `gigs_location` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `name` varchar(120)
)
;
CREATE TABLE `gigs_venue` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `name` varchar(120),
    `contact` varchar(250),
    `url` varchar(60)
)
;
CREATE TABLE `gigs_gig` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `date` date,
    `details` varchar(250),
    `location_id` integer NOT NULL,
    `venue_id` integer NOT NULL
)
;
ALTER TABLE `gigs_gig` ADD CONSTRAINT `venue_id_refs_id_3d901b6d` FOREIGN KEY (`venue_id`) REFERENCES `gigs_venue` (`id`);
ALTER TABLE `gigs_gig` ADD CONSTRAINT `location_id_refs_id_2f8d7a0` FOREIGN KEY (`location_id`) REFERENCES `gigs_location` (`id`);COMMIT;

I've triple checked the data, gone through to make sure all the relationships and data is ok after importing. But I'm still getting this error, three days on... I'm stuck with what to do about it. I can't imagine the "DeprecationWarning" is going to be a problem here. I really need to dump this data back out as JSON.

Many thanks for any help at all.

Finzer

Run it with --traceback to see the underlying error – Pacemaker

Did you managed to find the error?. I've ran in a similar situation trying to dump a bunch of models obtained via inspectdb... I do not get DoesNotExists though (may be due to the newer version). But django fails to lookup an existing coulmn... – Carrolcarroll

In my case, the problem was in non-standard models described for which there were no tables, even if such a model is indicated by managed=False. The solution is to exclude such models from the dump with the option --exclude App.Model – Toddy

Answer 1

Could be something similar to this.

Run it with:

python manage.py dumpdata --indent=2 -v 2 --traceback gigs 

To see the underlying error.

Pacemaker

Answer 2

I once ran in a similar problem where the error message was as mesmerizing as yours. The cause was a lack of memory on my server. It seems that generating dumps in json is quite memory expensive. I had only 60meg of memory (at djangohosting.ch) and it was not enough to get a dump for a mysql DB for which the mysql dump was only 1meg.

I was able to find out by watching the python process hit the 60meg limit using the top command in a second command line while running manage.py dumpdata in a first one.

My solution : get the mysql dump and then load it on my desktop pc, before generating the json dump. That said, for backup purposes, the mysql dumps are enough.

The command to get a mysql dump is the following :

mysqldump -p [password] -u [username] [database_name] > [dump_file_name].sql

That said, your problem could be completely different. You should really look at every table that has a foreign key to your Location table, and check if there is no field pointing to a previously deleted location. Unfortunately MySQL is very bad at maintaining Referential integrity, and you cannot count on it.

Shope

Answer 3

you can --exclude that particular app which is creating problem , still there will be database tables , it worked for me

python manage.py dumpdata > backedup_data.json --exclude app_name

Wisla

Answer 4

This error shows because there's a mismatch between your DB's schema and your Models.

You can try find it manually or you could just go ahead and install django-extensions

pip install django-extensions

and use the sqldiff command which will print you exactly wheres the problem.

python manage.py sqldiff -a -t

First and foremost, make your models match what your db has. Then run migrations and a fake migrate:

python manage.py makemigrations && python manage.py migrate --fake

That alone should let you run a dump. As soon as django makes sure the DB's schema matches your models, it will let you.

Moving forward, you can update your models and re-run the migrations as usual:

python manage.py makemigrations && python manage.py migrate

Psychoactive