GreenDAO schema update and data migration?
Asked Answered
A

7

60

I'm evaluating GreenDAO for consideration in a commercial Android app I will be working on and wanted to determine the migration path for schema updates.

Am I correct in asserting that I would have to write a custom OpenHelper which provides the onUpdate() and extracts transforms and stores data according to the new schema? This assumption raises some interesting questions around ordering of calls and partitioning of responsibility.

I have not been able to find any documentation around schema update and data migration for GreenDAO.

Here are a bunch of blog articles I have written on this topic:

  1. Review of greenDAO
  2. Part 1 – Schema Generation
  3. Part 2 – Schema Migration
  4. Part 3 – Testing Schema Migration
Alpheus answered 14/11, 2012 at 4:41 Comment(9)
I'm having the same issue. did you write this helper?Deport
I did, however the structure of these helpers is such that it is very tightly coupled to your solution (works directly on your tables and columns of your database in regards to the version changes) so it is not reusable; sorry.Alpheus
@Alpheus Can you tell us where you put the OpenHelper? At the moment the DaoMaster object extends the SQLiteOpenHelper but this file is generated and not meant to be edited. Or did you just extend the DaoMaster? thanksImmigration
@Tim, I created a new class called UpgradeHelper in my own package, e.g. "public class UpgradeHelper extends OpenHelper {". I am writing a 3 part blog article on schema generation, Migration and Migration testing with greenDAO due out soon. I will update this question when finished so please star it if you want to be notified.Alpheus
@Alpheus Ah ok. So you kinda ignore the OpenHelper in the DaoMaster completely. Thanks for the reply. Looking forward to your article.Immigration
@Immigration My custom UpgradeHelper extends DaoMaster.OpenHelper, then I use it like this. "OpenHelper helper = new UpgradeHelper(application, Constants.DB_NAME, null);"Alpheus
Hey @syntax, your links are not working, please provide working links, cheers.Tarsal
Sorry @MahdiAlkhatib the blog which hosted my solution has been closed and the hosted instructions and files have been taken down. Good news is that the web.archive still has copies; I have updated the links above, they should work :) EnjoyAlpheus
Thanks for providing those links. I think the way you manage the migrations is really good.Leupold
R
21

You assumed correctly. There is no change tracking between different schema versions today. Thus you need to write SQL yourself when doinng schema upgrades.

Representational answered 15/11, 2012 at 18:10 Comment(1)
i thing solutions proposed bellow are really great and it will be more then perfect if you decide to add them to greendao.Petrolatum
P
40

Thinking about the pleonasmik's approach (by the way, thanks, it was really helpfull), I created one MigrationHelper class.

How it works:

  1. The class catch all the Daos that you got
  2. Creates the temporary tables based on the old version's scheme (generateTempTables method)
  3. Import all the data to this new tables (generateTempTables method)
  4. Drop all the tables of the old version (DaoMaster.dropAllTables method)
  5. Creates the tables of the new version (DaoMaster.createAllTables method)
  6. Updates the new version's tables from the temporaries (restoreData method)
  7. Drop all temporary tables (restoreData method)

Migration Helper class:

import android.database.Cursor;
import android.database.sqlite.SQLiteDatabase;
import android.text.TextUtils;
import android.util.Log;

import com.crashlytics.android.Crashlytics;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import de.greenrobot.dao.AbstractDao;
import de.greenrobot.dao.internal.DaoConfig;
import greendao.DaoMaster;


/**
 * Created by pokawa on 18/05/15.
 */
public class MigrationHelper {

    private static final String CONVERSION_CLASS_NOT_FOUND_EXCEPTION = "MIGRATION HELPER - CLASS DOESN'T MATCH WITH THE CURRENT PARAMETERS";
    private static MigrationHelper instance;

    public static MigrationHelper getInstance() {
        if(instance == null) {
            instance = new MigrationHelper();
        }
        return instance;
    }

    public void migrate(SQLiteDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        generateTempTables(db, daoClasses);
        DaoMaster.dropAllTables(db, true);
        DaoMaster.createAllTables(db, false);
        restoreData(db, daoClasses);
    }

    private void generateTempTables(SQLiteDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        for(int i = 0; i < daoClasses.length; i++) {
            DaoConfig daoConfig = new DaoConfig(db, daoClasses[i]);

            String divider = "";
            String tableName = daoConfig.tablename;
            String tempTableName = daoConfig.tablename.concat("_TEMP");
            ArrayList<String> properties = new ArrayList<>();

            StringBuilder createTableStringBuilder = new StringBuilder();

            createTableStringBuilder.append("CREATE TABLE ").append(tempTableName).append(" (");

            for(int j = 0; j < daoConfig.properties.length; j++) {
                String columnName = daoConfig.properties[j].columnName;

                if(getColumns(db, tableName).contains(columnName)) {
                    properties.add(columnName);

                    String type = null;

                    try {
                        type = getTypeByClass(daoConfig.properties[j].type);
                    } catch (Exception exception) {
                        Crashlytics.logException(exception);
                    }

                    createTableStringBuilder.append(divider).append(columnName).append(" ").append(type);

                    if(daoConfig.properties[j].primaryKey) {
                        createTableStringBuilder.append(" PRIMARY KEY");
                    }

                    divider = ",";
                }
            }
            createTableStringBuilder.append(");");

            db.execSQL(createTableStringBuilder.toString());

            StringBuilder insertTableStringBuilder = new StringBuilder();

            insertTableStringBuilder.append("INSERT INTO ").append(tempTableName).append(" (");
            insertTableStringBuilder.append(TextUtils.join(",", properties));
            insertTableStringBuilder.append(") SELECT ");
            insertTableStringBuilder.append(TextUtils.join(",", properties));
            insertTableStringBuilder.append(" FROM ").append(tableName).append(";");

            db.execSQL(insertTableStringBuilder.toString());
        }
    }

    private void restoreData(SQLiteDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        for(int i = 0; i < daoClasses.length; i++) {
            DaoConfig daoConfig = new DaoConfig(db, daoClasses[i]);

            String tableName = daoConfig.tablename;
            String tempTableName = daoConfig.tablename.concat("_TEMP");
            ArrayList<String> properties = new ArrayList();

            for (int j = 0; j < daoConfig.properties.length; j++) {
                String columnName = daoConfig.properties[j].columnName;

                if(getColumns(db, tempTableName).contains(columnName)) {
                    properties.add(columnName);
                }
            }

            StringBuilder insertTableStringBuilder = new StringBuilder();

            insertTableStringBuilder.append("INSERT INTO ").append(tableName).append(" (");
            insertTableStringBuilder.append(TextUtils.join(",", properties));
            insertTableStringBuilder.append(") SELECT ");
            insertTableStringBuilder.append(TextUtils.join(",", properties));
            insertTableStringBuilder.append(" FROM ").append(tempTableName).append(";");

            StringBuilder dropTableStringBuilder = new StringBuilder();

            dropTableStringBuilder.append("DROP TABLE ").append(tempTableName);

            db.execSQL(insertTableStringBuilder.toString());
            db.execSQL(dropTableStringBuilder.toString());
        }
    }

    private String getTypeByClass(Class<?> type) throws Exception {
        if(type.equals(String.class)) {
            return "TEXT";
        }
        if(type.equals(Long.class) || type.equals(Integer.class) || type.equals(long.class)) {
            return "INTEGER";
        }
        if(type.equals(Boolean.class)) {
            return "BOOLEAN";
        }

        Exception exception = new Exception(CONVERSION_CLASS_NOT_FOUND_EXCEPTION.concat(" - Class: ").concat(type.toString()));
        Crashlytics.logException(exception);
        throw exception;
    }

    private static List<String> getColumns(SQLiteDatabase db, String tableName) {
        List<String> columns = new ArrayList<>();
        Cursor cursor = null;
        try {
            cursor = db.rawQuery("SELECT * FROM " + tableName + " limit 1", null);
            if (cursor != null) {
                columns = new ArrayList<>(Arrays.asList(cursor.getColumnNames()));
            }
        } catch (Exception e) {
            Log.v(tableName, e.getMessage(), e);
            e.printStackTrace();
        } finally {
            if (cursor != null)
                cursor.close();
        }
        return columns;
    }
}

And here's an example that how it should be called at the DaoMaster.java class:

@Override
    public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
        Log.i("greenDAO", "Upgrading schema from version " + oldVersion + " to " + newVersion + " by migrating all tables data");

        MigrationHelper.getInstance().migrate(db,
                UserDao.class,
                ItemDao.class);
    }
Pyroligneous answered 19/5, 2015 at 19:37 Comment(6)
Now that's an awesome code ! Thank you for sharing :)Seismography
I really like the solution a lot, it is short and powerful. One question I had though is that it doesn't seem like it would be able to handle the case where you add fields which are mandatory (required). There doesn't seem to be any point where you can provide default values etc for populating values in mandatory fields etc (since the migration code is generic), so unless I'm wrong this solution may limit the types of changes you can make on your database.Alpheus
I know that code published under Stack Overflow is under Creative Commons... But this guy copied the code over to his own GitHub repo without crediting this thread at all: github.com/yuweiguocn/GreenDaoUpgradeHelper/blob/master/library/…Franciskus
Hi guys, @EnricoSusatyo really thanks for warning me about this, but that's OK, I won't reclaim it, I really want it to be free and I don't want the credits as well. No worries, and I hope I helped you, guys, and thanks again, mate. :DPyroligneous
@PedroOkawa Do I need to set all my DAOs (the old and new ones) into the migration's method call to migrate my database correctly?Sldney
medium.com/@kidusmamuye/…Udo
R
21

You assumed correctly. There is no change tracking between different schema versions today. Thus you need to write SQL yourself when doinng schema upgrades.

Representational answered 15/11, 2012 at 18:10 Comment(1)
i thing solutions proposed bellow are really great and it will be more then perfect if you decide to add them to greendao.Petrolatum
W
12

This is the same code of @PedroOkawa that works with the GreenDao 3.+ with fixed errors:

import android.database.Cursor;
import android.database.sqlite.SQLiteDatabase;
import android.support.annotation.NonNull;
import android.text.TextUtils;

import org.greenrobot.greendao.AbstractDao;
import org.greenrobot.greendao.database.Database;
import org.greenrobot.greendao.database.StandardDatabase;
import org.greenrobot.greendao.internal.DaoConfig;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

/**
 * Createdby PedroOkawa and modified by MBH on 16/08/16.
 */
public final class MigrationHelper {

    public static void migrate(SQLiteDatabase sqliteDatabase, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        StandardDatabase db = new StandardDatabase(sqliteDatabase);
        generateNewTablesIfNotExists(db, daoClasses);
        generateTempTables(db, daoClasses);
        dropAllTables(db, true, daoClasses);
        createAllTables(db, false, daoClasses);
        restoreData(db, daoClasses);
    }

    public static void migrate(StandardDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        generateNewTablesIfNotExists(db, daoClasses);
        generateTempTables(db, daoClasses);
        dropAllTables(db, true, daoClasses);
        createAllTables(db, false, daoClasses);
        restoreData(db, daoClasses);
    }

    private static void generateNewTablesIfNotExists(StandardDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        reflectMethod(db, "createTable", true, daoClasses);
    }

    private static void generateTempTables(StandardDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        for (int i = 0; i < daoClasses.length; i++) {
            DaoConfig daoConfig = new DaoConfig(db, daoClasses[i]);
            String tableName = daoConfig.tablename;
            String tempTableName = daoConfig.tablename.concat("_TEMP");
            StringBuilder insertTableStringBuilder = new StringBuilder();
            insertTableStringBuilder.append("CREATE TEMP TABLE ").append(tempTableName);
            insertTableStringBuilder.append(" AS SELECT * FROM ").append(tableName).append(";");
            db.execSQL(insertTableStringBuilder.toString());
        }
    }

    private static void dropAllTables(StandardDatabase db, boolean ifExists, @NonNull Class<? extends AbstractDao<?, ?>>... daoClasses) {
        reflectMethod(db, "dropTable", ifExists, daoClasses);
    }

    private static void createAllTables(StandardDatabase db, boolean ifNotExists, @NonNull Class<? extends AbstractDao<?, ?>>... daoClasses) {
        reflectMethod(db, "createTable", ifNotExists, daoClasses);
    }

    /**
     * dao class already define the sql exec method, so just invoke it
     */
    private static void reflectMethod(StandardDatabase db, String methodName, boolean isExists, @NonNull Class<? extends AbstractDao<?, ?>>... daoClasses) {
        if (daoClasses.length < 1) {
            return;
        }
        try {
            for (Class cls : daoClasses) {
                Method method = cls.getDeclaredMethod(methodName, Database.class, boolean.class);
                method.invoke(null, db, isExists);
            }
        } catch (NoSuchMethodException e) {
            e.printStackTrace();
        } catch (InvocationTargetException e) {
            e.printStackTrace();
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        }
    }

    private static void restoreData(StandardDatabase db, Class<? extends AbstractDao<?, ?>>... daoClasses) {
        for (int i = 0; i < daoClasses.length; i++) {
            DaoConfig daoConfig = new DaoConfig(db, daoClasses[i]);
            String tableName = daoConfig.tablename;
            String tempTableName = daoConfig.tablename.concat("_TEMP");
            // get all columns from tempTable, take careful to use the columns list
            List<String> columns = getColumns(db, tempTableName);
            ArrayList<String> properties = new ArrayList<>(columns.size());
            for (int j = 0; j < daoConfig.properties.length; j++) {
                String columnName = daoConfig.properties[j].columnName;
                if (columns.contains(columnName)) {
                    properties.add(columnName);
                }
            }
            if (properties.size() > 0) {
                final String columnSQL = TextUtils.join(",", properties);

                StringBuilder insertTableStringBuilder = new StringBuilder();
                insertTableStringBuilder.append("INSERT INTO ").append(tableName).append(" (");
                insertTableStringBuilder.append(columnSQL);
                insertTableStringBuilder.append(") SELECT ");
                insertTableStringBuilder.append(columnSQL);
                insertTableStringBuilder.append(" FROM ").append(tempTableName).append(";");
                db.execSQL(insertTableStringBuilder.toString());
            }
            StringBuilder dropTableStringBuilder = new StringBuilder();
            dropTableStringBuilder.append("DROP TABLE ").append(tempTableName);
            db.execSQL(dropTableStringBuilder.toString());
        }
    }

    private static List<String> getColumns(StandardDatabase db, String tableName) {
        List<String> columns = null;
        Cursor cursor = null;
        try {
            cursor = db.rawQuery("SELECT * FROM " + tableName + " limit 0", null);
            if (null != cursor && cursor.getColumnCount() > 0) {
                columns = Arrays.asList(cursor.getColumnNames());
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            if (cursor != null)
                cursor.close();
            if (null == columns)
                columns = new ArrayList<>();
        }
        return columns;
    }

}

and the usage is:

@Override
public void onUpgrade(SQLiteDatabase sqLiteDatabase, int oldVersion, int newVersion) {
    MigrationHelper.migrate(new StandardDatabase(sqLiteDatabase),
                UserDao.class,
                ItemDao.class);
    // OR you can use it like this (Dont use both it is example of 2 different usages)
    MigrationHelper.migrate(sqLiteDatabase,
                UserDao.class,
                ItemDao.class);
}

StandardDatabase can be found in greendao and this is the import:

import org.greenrobot.greendao.database.StandardDatabase;

Thanks again to @PedroOkawa :)

Wapentake answered 16/8, 2016 at 20:41 Comment(0)
G
12

For those of you looking to update the database schema version on greenDAO 3, add this to your app's build.gradle file above dependencies:

apply plugin: 'org.greenrobot.greendao'

greendao {
    schemaVersion 1
}
Gullah answered 9/6, 2017 at 12:5 Comment(1)
It is worth remembering that this technique unfortunately deletes the local database and recreates it. Thanks.Catamaran
J
10

I think that my answer to a similiar question can help with this approach. If you really need to migrate data I suggest you that if you for example need to deal with some constrains changes or things that are not supported in SQLite you write the migration yourself. For instance an example of a migrator helper (following the approach I did on the linked response) could be:

public class DBMigrationHelper6 extends AbstractMigratorHelper {

/* Upgrade from DB schema 6 to schema 7 , version numbers are just examples*/

public void onUpgrade(SQLiteDatabase db) {

    /* Create a temporal table where you will copy all the data from the previous table that you need to modify with a non supported sqlite operation */
    db.execSQL("CREATE TABLE " + "'post2' (" + //
            "'_id' INTEGER PRIMARY KEY ," + // 0: id
            "'POST_ID' INTEGER UNIQUE ," + // 1: postId
            "'USER_ID' INTEGER," + // 2: userId
            "'VERSION' INTEGER," + // 3: version
            "'TYPE' TEXT," + // 4: type
            "'MAGAZINE_ID' TEXT NOT NULL ," + // 5: magazineId
            "'SERVER_TIMESTAMP' INTEGER," + // 6: serverTimestamp
            "'CLIENT_TIMESTAMP' INTEGER," + // 7: clientTimestamp
            "'MAGAZINE_REFERENCE' TEXT NOT NULL ," + // 8: magazineReference
            "'POST_CONTENT' TEXT);"); // 9: postContent

    /* Copy the data from one table to the new one */
    db.execSQL("INSERT INTO post2 (_id, POST_ID, USER_ID, VERSION, TYPE,  MAGAZINE_ID, SERVER_TIMESTAMP, CLIENT_TIMESTAMP, MAGAZINE_REFERENCE, POST_CONTENT)" +
            "   SELECT _id, POST_ID, USER_ID, VERSION, TYPE,  MAGAZINE_ID, SERVER_TIMESTAMP, CLIENT_TIMESTAMP, MAGAZINE_REFERENCE, POST_CONTENT FROM post;");

    /* Delete the previous table */
    db.execSQL("DROP TABLE post");
    /* Rename the just created table to the one that I have just deleted */
    db.execSQL("ALTER TABLE post2 RENAME TO post");

    /* Add Index/es if you want them */
    db.execSQL("CREATE INDEX " + "IDX_post_USER_ID ON post" +
            " (USER_ID);");

   }
}
Joyann answered 23/9, 2014 at 9:29 Comment(0)
S
1

Pedro Okawa' solution is correct, but you need to write your "UpgradeHelper" to extend "OpenHelper" because DaoMaster is overwritten each time you re-generate DAO code.

Example:

public class UpgradeHelper extends DaoMaster.OpenHelper {

public UpgradeHelper(Context context, String name, SQLiteDatabase.CursorFactory factory) {
    super(context, name, factory);
}

@Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {

    Log.i("greenDAO", "Upgrading schema from version " + oldVersion + " to " + newVersion + " by migrating all tables data");

    MigrationHelper.getInstance().migrate(db,
            UserDao.class,
            ItemDao.class,
            AnotherClassToGenerateDao.class);
}

}

Sumter answered 9/3, 2016 at 15:32 Comment(2)
how can I call this upgradeHelperSldney
You have to create a new class called "UpgradedHelper" and override that methodSumter
A
0

If you are just looking for a way to add new Tables to your schema without deleting your user's data and don't need to transform any existing data, take a look at my answer to this question for a discrete example of how to do it with greenDao.

Alaster answered 8/5, 2013 at 14:6 Comment(2)
Hi, thanks for your comment. If you read the blog posts added in edits in my question above you should find a much more comprehensive solution to this problem of migration etc than the one at the link provided. CheersAlpheus
I read all 4 of your blog posts before I even asked my question. Because the information that I was looking for was buried among the pattern that you're using, I thought I would provide a discrete example since there don't seem to be any on SO related to greenDao. I am actually using a pattern very similar to yours in my real app, but the question I posed was simply how to add a new table, not what pattern to use when upgrading your schema. I appreciate your posts, just wanted to provide a more succinct answer, as stated in the Disclaimer in my answer.Alaster

© 2022 - 2024 — McMap. All rights reserved.