Following backreferences of unknown kinds in NDB

Question

I'm in the process of writing my first RESTful web service atop GAE and the Python 2.7 runtime; I've started out using Guido's shiny new ndb API.

However, I'm unsure how to solve a particular case without the implicit back-reference feature of the original db API. If the user-agent requests a particular resource and those resources 1 degree removed:

host/api/kind/id?depth=2

What's the best way to discover a related collection of entities from the "one" in a one-to-many relationship, given that the kind of the related entity is unknown at development time?

• I'm unable to use a replacement query as described in a previous SO inquiry due to the latter restriction. The fact that my model is definable at runtime (and therefore isn't hardcoded) prevents me from using a query to filter properties for matching keys.

• Ancestor and other kindless queries are also out due to the datastore limitation that prevents me from filtering on a property without the kind specified.

Thus far, the only idea I've had (beyond reverting to the db api) is to use a cross-group transaction to write my own reference on the "one", either by updating an ndb.StringProperty(repeat=True) containing all the related kinds when an entity of a new kind is introduced or by simply maintaining a list of keys on the "one" ndb.KeyProperty(repeat=True) every time a related "many" entity is written to the datastore.

I'm hoping someone more experienced than myself can suggest a better approach.

Given jmort253's suggestion, I'll try to augment my question with a concrete example adapted from the docs:

class Contact(ndb.Expando):
    """ The One """

    # basic info
    name = ndb.StringProperty()
    birth_day = ndb.DateProperty()

    # If I were using db, a collection called 'phone_numbers' would be implicitly 
    # created here.  I could use this property to retrieve related phone numbers 
    # when this entity was queried.  Since NDB lacks this feature, the service 
    # will neither have a reference to query nor the means to know the 
    # relationship exists in the first place since it cannot be hard-coded.  The
    # data model is extensible and user-defined at runtime; most relationships
    # will be described only in the data, and must be discoverable by the server.
    # In this case, when Contact is queried, I need a way to retrieve the
    # collection of phone numbers.

    # Company info.
    company_title = ndb.StringProperty()
    company_name = ndb.StringProperty()
    company_description = ndb.StringProperty()
    company_address = ndb.PostalAddressProperty()

class PhoneNumber(ndb.Expando):
    """ The Many """

    # no collection_name='phone_numbers' equivalent exists for the key property
    contact = ndb.KeyProperty(kind='Contact')
    number = ndb.PhoneNumberProperty()

Albion

Hi, can you maybe show an example to help understand the problem a little better. Many people are visual, so seeing may help clarify your question. Good luck! :) – Dortch

If you don't know what entity kind you're looking for, what are you going to do with it/them when you fetch them? – Fining

The service layer will simply serialize the related entities it discovered into JSON representation of the explicitly requested entity. It's up to the client application to determine how they're used; the server doesn't care. – Albion

Answer 1

Interesting question! So basically you want to look at the Contact class and find out if there is some other model class that has a KeyProperty referencing it; in this example PhoneNumber (but there could be many).

I think the solution is to ask your users to explicitly add this link when the PhoneNumber class is created.

You can make this easy for your users by giving them a subclass of KeyProperty that takes care of this; e.g.

class LinkedKeyProperty(ndb.KeyProperty):
    def _fix_up(self, cls, code_name):
        super(LinkedKeyProperty, self)._fix_up(cls, code_name)
        modelclass = ndb.Model._kind_map[self._kind]
        collection_name = '%s_ref_%s_to_%s' % (cls.__name__,
                                               code_name,
                                               modelclass.__name__)
        setattr(modelclass, collection_name, (cls, self))

Exactly how you pick the name for the collection and the value to store there is up to you; just put something there that makes it easy for you to follow the link back. The example would create a new attribute on Contact:

Contact.PhoneNumber_ref_contact_to_Contact == (PhoneNumber, PhoneNumber.contact)

[edited to make the code working and to add an example. :-) ]

Preschool

Many thanks for the post, and even more for the unsolicited elaboration. B) Your synopsis of the problem is spot-on. This also appears to be right approach. I'm slowly beginning to acquire an understanding of the mechanics behind it as I scour the package source; however, self._kind, which corresponds to the kwarg passed to the LinkedKeyProperty(kind='Contact') constructor, doesn't exist on the kind map for some reason - at least not in my implementation. In the case of the example, I assume this should be retrieving a reference to the Contact model. – Albion

ndb.Model._kind_map[self._kind] == *crash* – Albion

If you write LinkedKeyProperty exactly as I showed it should work. Perhaps you didn't derive it from ndb.KeyProperty? Or you didn't make the super() call first? If you can't get it to work, please show the full code you tried, and the full traceback. – Preschool

It works perfectly. The error was clearly mine, arising from some inexplicable change to application state that had nothing to do with the source. I attempted to reproduce this error a few times in search of a better explanation, but concluded time was best spent making new features work instead of explaining why one didn't. – Albion

Answer 2

Sound like a good use case for ndb.StructuredProperty.

Scharf

Thanks for the suggestion! However, ndb.StructuredProperty has its own disqualifying limitation in my case: it "cannot be retrieved independently of the Contact entity to which [it] belongs". – Albion