Maintain uniqueness of a property in the NDB database

Question

An NDB model contains two properties: email and password. How to avoid adding to the database two records with the same email? NDB doesn't have UNIQUE option for a property, like relational databases do.

Checking that new email is not in the database before adding—won't satisfy me, because two parallel processes can both simultaneously do the checking and each add the same email.

I'm not sure that transactions can help here, I am under this impression after reading some of the manuals. Maybe the synchronous transactions? Does it mean one at a time?

Commemorative

Answer 1

Create the key of the entity by email, then use get_or_insert to check if exists.

Also read about keys , entities. and models

#ADD
key_a = ndb.Key(Person, email);
person = Person(key=key_a)
person.put()

#Insert unique    
a = Person.get_or_insert(email)

or if you want to just check

#ADD
key_a = ndb.Key(Person, email);
person = Person(key=key_a)
person.put()

#Check if it's added
new_key_a =ndb.Key(Person, email);
a = new_key_a.get()
if a is not None:
    return

Take care. Changing email will be really difficult (need to create new entry and copy all entries to new parent).

For that thing maybe you need to store the email, in another entity and have the User be the parent of that.

Another way is to use Transactions and check the email property. Transaction's work in the way: First that commits is the First that wins. A concept which means that if 2 users check for email only the first (lucky) one will succeed, thus your data will be consistent.

Krasnoyarsk

Using get_or_insert() I won't be sure: either I have added a new record or I've got the existing one. – Commemorative

changing email will be pain in the a**... or better.. not possible :) So for emails it's not a good solution,. – Crisis

@Graduate yes should be key_a fixed that. – Krasnoyarsk

@Crisis Yes it will be a big pain in the a**. – Krasnoyarsk

I've got AttributeError: 'Key' object has no attribute 'put' – Commemorative

And mustn't I use repr(key) for this function Person.get_or_insert(repr(new_key_a)) – Commemorative

@Graduate Read more about keys. Check updated asnwer for you error. – Krasnoyarsk

@Graduate repr is not required. Comprehend first the concept of entities and keys (this is not sql) and then you will find that uniqueity is not that difficult to accomplish – Krasnoyarsk

I don't like to correct you, but the other people will rely on your answer, so sorry. It should be Person.get_or_insert(email), just email, not the whole key like in your code (you give ndb.Key(Person, email)). The method get_or_insert is called on the class Person so it gets the first part of the key from it. (I have checked it in real code.) – Commemorative

Thanks forgot about that. Yes indeed it's the key_name. You may as well edit the answer your self. Difficult to test and work at the same time. Nevertheless I hope you are getting where you need. @Graduate – Krasnoyarsk

Answer 2

Maybe you are looking for the webapp2-authentication module, that can handle this for you. It can be imported like this import webapp2_extras.appengine.auth.models. Look here for a complete example.

Remaremain

Answer 3

I also ran into this problem, and the solution above didn't solve my problem:

• making it a key was unacceptable in my case (i need the property to be changeable in the future)
• using transactions on the email property doesn't work AFAIK (you can't do queries on non-key names inside transactions, so you can't check whether the e-mail already exists).

I ended up creating a separate model with no properties, and the unique property (email address) as the key name. In the main model, I store a reference to the email model (instead of storing the email as a string). Then, I can make 'change_email' a transaction that checks for uniqueness by looking up the email by key.

Thumping

Answer 4

This is something that I've come across as well and I settled on a variation of @Remko's solution. My main issue with checking for an existing entity with the given email is a potential race condition like op stated. I added a separate model that uses an email address as the key and has a property that holds a token. By using get_or_insert, the returned entities token can be checked against the token passed in and if they match then the model was inserted.

import os
from google.appengine.ext import ndb

class UniqueEmail(ndb.Model):
    token = ndb.StringProperty()

class User(ndb.Model):
    email = ndb.KeyProperty(kind=UniqueEmail, required=True)
    password = ndb.StringProperty(required=True)

def create_user(email, password):
    token = os.urandom(24)
    unique_email = UniqueEmail.get_or_insert(email,
                                             token=token)

    if token == unique_email.token:
        # If the tokens match, that means a UniqueEmail entity
        # was inserted by this process.
        # Code to create User goes here.
    # The tokens do not match, therefore the UniqueEmail entity
    # was retrieved, so the email is already in use.
    raise ValueError('That user already exists.')

Nadaba

Answer 5

I implemented a generic structure to control unique properties. This solution can be used for several kinds and properties. Besides, this solution is transparent for other developers, they use NDB methods put and delete as usual.

1) Kind UniqueCategory: a list of unique properties in order to group information. Example:

‘User.nickname’

2) Kind Unique: it contains the values of each unique property. The key is the own property value which you want to control of. I save the urlsafe of the main entity instead of the key or key.id() because is more practical and it doesn’t have problem with parent and it can be used for different kinds. Example:

parent: User.nickname
key: AVILLA
reference_urlsafe: ahdkZXZ-c3RhcnQtb3BlcmF0aW9uLWRldnINCxIEVXNlciIDMTIzDA (User key)

3) Kind User: for instance, I want to control unique values for email and nickname. I created a list called ‘uniqueness’ with the unique properties. I overwritten method put in transactional mode and I wrote the hook _post_delete_hook when one entity is deleted.

4) Exception ENotUniqueException: custom exception class raised when some value is duplicated.

5) Procedure check_uniqueness: check whether a value is duplicated.

6) Procedure delete_uniqueness: delete unique values when the main entity is deleted.

Any tips or improvement are welcome.

---

class UniqueCategory(ndb.Model):
    # Key = [kind name].[property name]

---

class Unique(ndb.Model):
    # Parent = UniqueCategory
    # Key = property value
    reference_urlsafe = ndb.StringProperty(required=True)

---

class ENotUniqueException(Exception):
    def __init__(self, property_name):
        super(ENotUniqueException, self).__init__('Property value {0} is duplicated'.format(property_name))

        self. property_name = property_name

---

class User(ndb.Model):
    # Key = Firebase UUID or automatically generated
    firstName = ndb.StringProperty(required=True)
    surname = ndb.StringProperty(required=True)
    nickname = ndb.StringProperty(required=True)
    email = ndb.StringProperty(required=True)

    @ndb.transactional(xg=True)
    def put(self):
        result = super(User, self).put()
        check_uniqueness (self)
        return result

    @classmethod
    def _post_delete_hook(cls, key, future):
        delete_uniqueness(key)

    uniqueness = [nickname, email]

---

def check_uniqueness(entity):
    def get_or_insert_unique_category(qualified_name):
        unique_category_key = ndb.Key(UniqueCategory, qualified_name)
        unique_category = unique_category_key.get()
        if not unique_category:
           unique_category = UniqueCategory(id=qualified_name)
           unique_category.put()

        return unique_category_key

    def del_old_value(key, attribute_name, unique_category_key):
        old_entity = key.get()
        if old_entity:
           old_value = getattr(old_entity, attribute_name)
           if old_value != new_value:
               unique_key = ndb.Key(Unique, old_value, parent=unique_category_key)
               unique_key.delete()

    # Main flow
    for unique_attribute in entity.uniqueness:
        attribute_name = unique_attribute._name
        qualified_name = type(entity).__name__ + '.' + attribute_name
        new_value = getattr(entity, attribute_name)

        unique_category_key = get_or_insert_unique_category(qualified_name)
        del_old_value(entity.key, attribute_name, unique_category_key)

        unique = ndb.Key(Unique, new_value, parent=unique_category_key).get()

        if unique is not None and unique.reference_urlsafe != entity.key.urlsafe():
           raise ENotUniqueException(attribute_name)
        else:
           unique = Unique(parent=unique_category_key,
                           id=new_value, 
                           reference_urlsafe=entity.key.urlsafe())
           unique.put()

---

def delete_uniqueness(key):
    list_of_keys = Unique.query(Unique.reference_urlsafe == key.urlsafe()).fetch(keys_only=True)

    if list_of_keys:
        ndb.delete_multi(list_of_keys)

Anarchist