db.ReferenceProperty() vs ndb.KeyProperty in App Engine

Question

ReferenceProperty was very helpful in handling references between two modules. Fox example:

class UserProf(db.Model):
    name = db.StringProperty(required=True)

class Team(db.Model):
    manager_name = db.ReferenceProperty(UserProf, collection_name='teams')
    name = db.StringProperty(required=True)

• To get 'manager_name' with team instance, we use team_ins.manager_name.
• To get 'teams' which are managed by particular user instance, we use user_instance.teams and iterate over.

Doesn't it look easy and understandable?

In doing same thing using NDB, we have to modify

db.ReferenceProperty(UserProf, collection_name='teams') --> ndb.KeyProperty(kind=UserProf)

• team_ins.manager_name.get() would give you manager name

• To get all team which are manger by particular user, we have to do

  for team in Team.query(Team.manager_name == user_ins.key): 
      print "team  name:", team.name
  

As you can see handling these kind of scenarios looks easier and readable in db than ndb.

• What is the reason for removing ReferenceProperty in ndb?
• Even db's query user_instance.teams would have doing the same thing as it is done in ndb's for loop. But in ndb, we are explicitly mentioning using for loop.
• What is happening behind the scenes when we do user_instance.teams?

Thanks in advance..

Sting

Answer 1

Tim explained it well. We found that a common anti-pattern was using reference properties and loading them one at a time, because the notation "entity.property1.property2" doesn't make it clear that the first dot causes a database "get" operation. So we made it more obvious by forcing you to write "entity.property1.get().property2", and we made it easier to do batch prefetching (without the complex solution from Nick's blog) by simply saying "entity.property1.get_async()" for a bunch of entities -- this queues a single batch get operation without blocking for the result, and when you next reference any of these properties using "entity.property1.get().property2" this won't start another get operation but just waits for that batch get to complete (and the second time you do this, the batch get is already complete). Also this way in-process and memcache integration comes for free.

Wyoming

Thanks. ndb is same as db but it is explicit and user will get to know that there is get() operation. There is no performance improvement in this case. I have read about ndb async, haven't used it yet. – Sting

@Tim and Guido, Can you please suggest any sample applications written using ndb similar to the one in code.google.com/p/google-app-engine-samples. And also best practices to follow in ndb based applications. It would help a lot. Thanks. – Sting

Calling entity.keyproperty.get().somevalue1 will fetch the entity referenced by keyproperty first time, in subsequent entity.keyproperty.get().somevalue2 and entity.keyproperty.get().somevalue3 will not make the db call ??? – Hemorrhoidectomy

Okay....Since the context is same (in same request and if no new thread is initialized), the entity is saved in in-context cache when get is called first time and same entity will be returned in subsequent calls to get function, so No new db calls. see this link (cloud.google.com/appengine/docs/python/ndb/cache#context) for more about in-context cache. – Hemorrhoidectomy

Answer 2

I don't know the answer as to why Guido didn't implement reference property.

However I found a spent a lot of time using pre_fetch_refprops http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine (pre fetches all of the reference properties by grabbing all the keys with get_value_for_datastore,) and then it does a get_multi on the keys.

This was vastly more efficient.

Also if the object referenced doesn't exist you would get an error when trying to fetch the object.

If you pickled an object which had references you ended up pickling a lot more than you probably planned too.

So I found except for the one case, where you have single entity and you wanted to grab the referenced object with .name type accessor you had to jump through all sorts of hoops to prevent the referenced entity from being fetched.

Ministration

Thanks. Hoping Guido himself will answer this question.!:) – Sting