Migrating existing auth.User data to new Django 1.5 custom user model?
Asked Answered
L

5

34

I'd prefer not to destroy all the users on my site. But I want to take advantage of Django 1.5's custom pluggable user model. Here's my new user model:

class SiteUser(AbstractUser):
    site = models.ForeignKey(Site, null=True)

Everything works with my new model on a new install (I've got other code, along with a good reason for doing this--all of which are irrelevant here). But if I put this on my live site and syncdb & migrate, I'll lose all my users or at least they'll be in a different, orphaned table than the new table created for my new model.

I'm familiar with South, but based on this post and some trials on my part, it seems its data migrations are not currently a fit for this specific migration. So I'm looking for some way to either make South work for this or for some non-South migration (raw SQL, dumpdata/loaddata, or otherwise) that I can run on each of my servers (Postgres 9.2) to migrate the users once the new table has been created while the old auth.User table is still in the database.

Lipp answered 15/2, 2013 at 21:59 Comment(0)
A
49

South is more than able to do this migration for you, but you need to be smart and do it in stages. Here's the step-by-step guide: (This guide presupposed you subclass AbstractUser, not AbstractBaseUser)

  1. Before making the switch, make sure that south support is enabled in the application that contains your custom user model (for the sake of the guide, we'll call it accounts and the model User). At this point you should not yet have a custom user model.

    $ ./manage.py schemamigration accounts --initial
    Creating migrations directory at 'accounts/migrations'...
    Creating __init__.py in 'accounts/migrations'...
    Created 0001_initial.py.
    
    $ ./manage.py migrate accounts [--fake if you've already syncdb'd this app]
     Running migrations for accounts:
     - Migrating forwards to 0001_initial.
     > accounts:0001_initial
     - Loading initial data for accounts.
    
  2. Create a new, blank user migration in the accounts app.

    $ ./manage.py schemamigration accounts --empty switch_to_custom_user
    Created 0002_switch_to_custom_user.py.
    
  3. Create your custom User model in the accounts app, but make sure it is defined as:

    class SiteUser(AbstractUser): pass
    
  4. Fill in the blank migration with the following code.

    # encoding: utf-8
    from south.db import db
    from south.v2 import SchemaMigration
    
    class Migration(SchemaMigration):
    
        def forwards(self, orm):
            # Fill in the destination name with the table name of your model
            db.rename_table('auth_user', 'accounts_user')
            db.rename_table('auth_user_groups', 'accounts_user_groups')
            db.rename_table('auth_user_user_permissions', 'accounts_user_user_permissions')
    
        def backwards(self, orm):
            db.rename_table('accounts_user', 'auth_user')
            db.rename_table('accounts_user_groups', 'auth_user_groups')
            db.rename_table('accounts_user_user_permissions', 'auth_user_user_permissions')
    
        models = { ....... } # Leave this alone
    
  5. Run the migration

    $ ./manage.py migrate accounts
     - Migrating forwards to 0002_switch_to_custom_user.
     > accounts:0002_switch_to_custom_user
     - Loading initial data for accounts.
    
  6. Make any changes to your user model now.

    # settings.py
    AUTH_USER_MODEL = 'accounts.User'
    
    # accounts/models.py
    class SiteUser(AbstractUser):
        site = models.ForeignKey(Site, null=True)
    
  7. create and run migrations for this change

    $ ./manage.py schemamigration accounts --auto
     + Added field site on accounts.User
    Created 0003_auto__add_field_user_site.py.
    
    $ ./manage.py migrate accounts
     - Migrating forwards to 0003_auto__add_field_user_site.
     > accounts:0003_auto__add_field_user_site
     - Loading initial data for accounts.
    

Honestly, If you already have good knowledge of your setup and already use south, It should be as simple as adding the following migration to your accounts module.

# encoding: utf-8
from south.db import db
from south.v2 import SchemaMigration
from django.db import models

class Migration(SchemaMigration):

    def forwards(self, orm):
        # Fill in the destination name with the table name of your model
        db.rename_table('auth_user', 'accounts_user')
        db.rename_table('auth_user_groups', 'accounts_user_groups')
        db.rename_table('auth_user_permissions', 'accounts_user_permissions')
        # == YOUR CUSTOM COLUMNS ==
        db.add_column('accounts_user', 'site_id',
            models.ForeignKey(orm['sites.Site'], null=True, blank=False)))

    def backwards(self, orm):
        db.rename_table('accounts_user', 'auth_user')
        db.rename_table('accounts_user_groups', 'auth_user_groups')
        db.rename_table('accounts_user_user_permissions', 'auth_user_user_permissions')
        # == YOUR CUSTOM COLUMNS ==
        db.remove_column('accounts_user', 'site_id')

    models = { ....... } # Leave this alone

EDIT 2/5/13: added rename for auth_user_group table. FKs will auto update to point at the correct table due to db constraints, but M2M fields' table names are generated from the names of the 2 end tables and will need manual updating in this manner.

EDIT 2: Thanks to @Tuttle & @pix0r for the corrections.

Antiphonal answered 25/2, 2013 at 2:49 Comment(16)
Enjoy your bounty! I ended up doing this with dumpdata/loaddata (and manipulating the dumped data in between) and it worked simply but this would have certainly worked had I had this understanding a bit earlier.Lipp
This didn't work for me. The migration containing the table rename simply does nothing, silently succeeding.Gareth
This worked great, but now schemamigration accounts --auto is trying to create the accounts_user table (which already exists...?), create new M2M tables for groups and user_permissions (which I can similarly rename), and update all the FKs I have pointing at User. Inspecting the constraints in postgres directly, all the FKs seem to have been migrated with the rename_table. Any way to tell south not to bother with these, or at least not try to create the new tables?Babcock
It was a "problem" with the frozen orm from 0002 -- it needs an 'accounts.user' entry so south won't think that model is new for the next migration. I did the schemamigration --auto, copied the 'accounts.user' entry from 0003 to 0002, deleted 0003 and re-created it, now all it wants to do is update the FKs. +1Babcock
Thanks for the great explanation! One thing: for the blank (empty) migration, the option is --empty. The --blank option throws an error.Ottavia
I get the error DatabaseError: near ")": syntax error when trying to run the migration in step 7. Any idea what could cause this?Mellott
The SQL that's failing is: INSERT INTO "_south_new_app_model" () SELECT FROM "app_model";Mellott
@TomMedley what does south generate for the migration in step 7 for you?Antiphonal
aren't auth_user_groups and auth_user_permissions to be migrated too?Bosky
@RiccardoGalli nope, just the Auth model. because we're using a database level rename operation, all the FKs will be updated to point at the right place.Antiphonal
@Antiphonal Having followed these instructions, I'm getting an error, see my question. Although the app works fine, I can't access the Users settings page.Mellott
@Antiphonal Incidentally, the previous error was to do with this not working with sqlite.Mellott
I think the example still misses renaming the auth_user_user_permissions table. Anyway it's the best 1.5 user model migration guide I found so far.Pinite
@eternicode: Hint, the copy-from 0003 creation can be generated with --stdout so no real file is created and does not need to be deleted.Pinite
@Pinite you are correct, I just went through this process myself.. I hope the author won't mind if I make the correction (*_user_permissions => *_user_user_permission). And @Antiphonal thank you very much for such an informative and accurate answer.Triecious
I know this is very old but I'm working on a similar migration currently. Is AUTH_USER_MODEL = 'accounts.User' really correct when the class is actually called SiteUser?Helmuth
S
16

My incredibly lazy way of doing this:

  1. Create a new model (User), extending AbstractUser. Within new model, in it's Meta, override db_table and set to 'auth_user'.

  2. Create an initial migration using South.

  3. Migrate, but fake the migration, using --fake when running migrate.

  4. Add new fields, create migration, run it normally.

This is beyond lazy, but works. You now have a 1.5 compliant User model, which just uses the old table of users. You also have a proper migration history.

You can fix this later on with manual migrations to rename the table.

Stethoscope answered 25/2, 2013 at 6:47 Comment(2)
I like this way! But I got a message from my ./manage.py migrate --fake that The following content types are stale and need to be deleted: auth | user Any objects related to these content types by a foreign key will also be deleted. Are you sure you want to delete these content types? If you're unsure, answer 'no'. Type 'yes' to continue, or 'no' to cancel: What did you enter?Chas
If you have any GenericForeignKeys, or if you use django's permissions (or anything else that FKs a ContentType), you'll need to add a migration that renames the appropriate ContentType instance to point to the new model.Antitank
B
4

I think you've correctly identified that a migration framework like South is the right way to go here. Assuming you're using South, you should be able to use the Data Migrations functionality to port the old users to your new model.

Specifically, I would add a forwards method to copy all rows in your user table to the new table. Something along the lines of:

def forwards(self, orm):
    for user in orm.User.objects.all():
        new_user = SiteUser(<initialize your properties here>)
        new_user.save()

You could also use the bulk_create method to speed things up.

Brendon answered 15/2, 2013 at 22:11 Comment(2)
One critical note: If you try to access auth.models.User after you've set AUTH_USER_MODEL to be your new user model, you'll get: AttributeError: Manager isn't available; User has been swapped for 'users.TenantSiteUser'. I'm going to try to do the migration and then set AUTH_USER_MODEL and see if that works.Lipp
@BenRoberts, In retrospect, there will likely be deeper problems not addressed above if you have any ForeignKeys pointing to the User model. You might be able to do two migrations though--one to create the new model and copy data, and then a second to set the new model as your User and then updated foreign keys. In the end, it may actually be easier to create and populate the new DB table in raw SQL, though that should be pretty easy with an INSERT...SELECT statement.Chengteh
L
3

I got tired of struggling with South so I actually ended up doing this differently and it worked out nicely for my particular situation:

First, I made it work with ./manage.py dumpdata, fixing up the dump, and then ./manage.py loaddata, which worked. Then I realized I could do basically the same thing with a single, self-contained script that only loads necessary django settings and does the serialization/deserialization directly.

Self-contained python script

## userconverter.py ##

import json
from django.conf import settings

settings.configure(
    DATABASES={ 
            # copy DATABASES configuration from your settings file here, or import it directly from your settings file (but not from django.conf.settings) or use dj_database_url
            },
    SITE_ID = 1, # because my custom user implicates contrib.sites (which is why it's in INSTALLED_APPS too)
    INSTALLED_APPS = ['django.contrib.sites', 'django.contrib.auth', 'myapp'])

# some things you have to import after you configure the settings
from django.core import serializers
from django.contrib.auth.models import User

# this isn't optimized for huge amounts of data -- use streaming techniques rather than loads/dumps if that is your case
old_users = json.loads(serializers.serialize('json', User.objects.all()))
for user in old_users:
    user['pk'] = None
    user['model'] = "myapp.siteuser"
    user['fields']["site"] = settings['SITE_ID']

for new_user in serializers.deserialize('json', json.dumps(old_users)):
    new_user.save()

With dumpdata/loaddata

I did the following:

1) ./manage.py dumpdata auth.User

2) Script to convert auth.user data to new user. (or just manually search and replace in your favorite text editor or grep) Mine looked something like:

def convert_user_dump(filename, site_id):
    file = open(filename, 'r')
    contents = file.read()
    file.close()
    user_list = json.loads(contents)
    for user in user_list:
        user['pk'] = None  # it will auto-increment
        user['model'] = "myapp.siteuser"
        user['fields']["site"] = side_id
    contents = json.dumps(user_list)
    file = open(filename, 'w')
    file.write(contents)
    file.close()

3) ./manage.py loaddata filename

4) set AUTH_USER_MODEL

*Side Note: One critical part of doing this type of migration, regardless of which technique you use (South, serialization/modification/deserialization, or otherwise) is that as soon as you set AUTH_USER_MODEL to your custom model in the current settings, django cuts you off from auth.User, even if the table still exists.*

Lipp answered 25/2, 2013 at 17:33 Comment(0)
O
2

We decided to switch to a custom user model in our Django 1.6/Django-CMS 3 project, perhaps a little bit late because we had data in our database that we didn't want to lose (some CMS pages, etc).

After we switched AUTH_USER_MODEL to our custom model, we had a lot of problems that we hadn't anticipated, because a lot of other tables had foreign keys to the old auth_user table, which wasn't deleted. So although things appeared to work on the surface, a lot of things broke underneath: publishing pages, adding images to pages, adding users, etc. because they tried to create an entry in a table that still had a foreign key to auth_user, without actually inserting a matching record into auth_user.

We found a quick and dirty way to rebuild all the tables and relations, and copy our old data across (except for users):

  • do a full backup of your database with mysqldump
  • do another backup with no CREATE TABLE statements, and excluding a few tables that won't exist after the rebuild, or will be populated by syncdb --migrate on a fresh database:
    • south_migrationhistory
    • auth_user
    • auth_user_groups
    • auth_user_user_permissions
    • auth_permission
    • django_content_types
    • django_site
    • any other tables that belong to apps that you removed from your project (you might only find this out by experimenting)
  • drop the database
  • recreate the database (e.g. manage.py syncdb --migrate)
  • create a dump of the empty database (to make it faster to go round this loop again)
  • attempt to load the data dump that you created above
  • if it fails to load because of a duplicate primary key or a missing table, then:
    • edit the dump with a text editor
    • remove the statements that lock, dump and unlock that table
    • reload the empty database dump
    • try to load the data dump again
    • repeat until the data dump loads without errors

The commands that we ran (for MySQL) were:

mysqldump <database> > ~/full-backup.sql
mysqldump <database> \
    --no-create-info \
    --ignore-table=<database>.south_migrationhistory \
    --ignore-table=<database>.auth_user \
    --ignore-table=<database>.auth_user_groups \
    --ignore-table=<database>.auth_user_user_permissions \
    --ignore-table=<database>.auth_permission \
    --ignore-table=<database>.django_content_types \
    --ignore-table=<database>.django_site \
> ~/data-backup.sql

./manage.py sqlclear
./manage.py syncdb --migrate
mysqldump <database> > ~/empty-database.sql

./manage.py dbshell < ~/data-backup.sql

(edit ~/data-backup.sql to remove data dumped from a table that no longer exists)

./manage.py dbshell < ~/empty-database.sql
./manage.py dbshell < ~/data-backup.sql

(repeat until clean)
Oecology answered 13/8, 2014 at 17:18 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.