How to delete a column (attribute) from a GAE datastore?

Question

I have a persistent class stored in a GAE datastore. I have removed one of the attributes from the class. The new records in this table show a value <none> for the removed attribute. But is there a way I can completely drop this column off the table?

Thanks.

Added the following 'migration' code according to moraes' suggestion, but it isn't achieving the desired result:

PersistenceManager pm = PMF.get().getPersistenceManager();
try {
    Query q = pm.newQuery(UserLogin.class);
    Collection<UserLogin> list = (Collection<UserLogin>) q.execute();

    Iterator<UserLogin> iter = list.iterator();
    while (iter.hasNext()) {
        UserLogin obj = (UserLogin) iter.next();
        obj.setLoginDate(obj.getLoginDate());
    }

    pm.makePersistentAll(list); 

} finally {
    pm.close();
}

Haymaker

Answer 1

I found the answer to this problem in this Article: http://code.google.com/appengine/articles/update_schema.html

"Removing Deleted Properties from the Datastore

If you remove a property from your model, you will find that existing entities still have the property. It will still be shown in the admin console and will still be present in the datastore. To really clean out the old data, you need to cycle through your entities and remove the data from each one.

• Make sure you have removed the properties from the model definition.

• If your model class inherits from db.Model, temporarily switch it to inherit from db.Expando. (db.Model instances can't be modified dynamically, which is what we need to do in the next step.)

• Cycle through existing entities (like described above). For each entity, use delattr to delete the obsolete property and then save the entity.

• If your model originally inherited from db.Model, don't forget to change it back after updating all the data."

And here is an example with code: http://sandrylogan.wordpress.com/2010/12/08/delattr/

Hedonics

Answer 2

If you are using ndb (and you probably should), you can easily delete properties by deleting them from entity._properties:

for entity in MyModel.query():
    if 'old_property' in entity._values:
        del entity._properties['old_property']
        del entity._values['old_property']
        entity.put()

Or you could make it faster by using an asynchronous query map:

@ndb.tasklet
def cleanup(entity):
    if 'old_property' in entity._values:
        del entity._properties['old_property']
        del entity._values['old_property']
        yield entity.put_async()

MyModel.query().map(cleanup)

Hinduism

hey, I think the "delete"s should instead be "del"s, but this worked great for me, thanks. – Hedonics

Answer 3

There is no concept of "table" in datastore. Each entity can have arbitrary properties that don't follow a common schema. The only "schema" is in your model code, and existing records don't change automatically when you change your models.

So, to delete the property from existing records, you need to iterate over all records and re-save them without the property.

Synergist

moraes - thanks for your help. Please refer to the 'migration' code in my OP. My UserLogin class is the persistent class and the unwanted field has been removed from it. But somehow I'm still seeing <none> under the old entries in the datastore. Did I miss something? – Haymaker

You need to call pm.makePersistentAll() passing the changed entities, to save them and effectively exclude the extra property that doesn't exist anymore. – Synergist

I added that call to the method, but there was no difference. See the updated code in my OP. I still see <none> for the removed property's values. Thanks. – Haymaker

BTW, I thought pm.makePersistentAll() would not be required as the changes would automatically be saved with pm.close() call. Anyway, either way the result is the same. – Haymaker

Answer 4

The datastore viewer gets its list of columns from the datastore stats, which are updated on a regular basis. If you've removed that column from every entity that had it, wait a day or two and the datastore viewer will stop showing it.

Organism