AppEngine: Query datastore for records with <missing> value

Question

I created a new property for my db model in the Google App Engine Datastore.

Old:

class Logo(db.Model):
  name = db.StringProperty()
  image = db.BlobProperty()

New:

class Logo(db.Model):
  name = db.StringProperty()
  image = db.BlobProperty()
  is_approved = db.BooleanProperty(default=False)

How to query for the Logo records, which to not have the 'is_approved' value set? I tried

logos.filter("is_approved = ", None)

but it didn't work. In the Data Viewer the new field values are displayed as .

Imphal

Answer 1

According to the App Engine documentation on Queries and Indexes, there is a distinction between entities that have no value for a property, and those that have a null value for it; and "Entities Without a Filtered Property Are Never Returned by a Query." So it is not possible to write a query for these old records.

A useful article is Updating Your Model's Schema, which says that the only currently-supported way to find entities missing some property is to examine all of them. The article has example code showing how to cycle through a large set of entities and update them.

Methenamine

Marzia from Google has confirmed there is no way to query for missing values. – Katzen

Answer 2

A practice which helps us is to assign a "version" field on every Kind. This version is set on every record initially to 1. If a need like this comes up (to populate a new or existing field in a large dataset), the version field allows iteration through all the records containing "version = 1". By iterating through, setting either a "null" or another initial value to the new field, bump the version to 2, store the record, allows populating the new or existing field with a default value.

The benefit to the "version" field is that the selection process can continue to select against that lower version number (initially set to 1) over as many sessions or as much time is needed until ALL records are updated with the new field default value.

Ecchymosis

Answer 3

Maybe this has changed, but I am able to filter records based on null fields.

When I try the GQL query SELECT * FROM Contact WHERE demo=NULL, it returns only records for which the demo field is missing.

According to the doc http://code.google.com/appengine/docs/python/datastore/gqlreference.html:

The right-hand side of a comparison can be one of the following (as appropriate for the property's data type): [...] a Boolean literal, as TRUE or FALSE; the NULL literal, which represents the null value (None in Python).

I'm not sure that "null" is the same as "missing" though : in my case, these fields already existed in my model but were not populated on creation. Maybe Federico you could let us know if the NULL query works in your specific case?

Impinge

The field wasn't missing if it was part of your model at the time the entity was created. And it was populated on creation, even if it was populated with None or the empty string. – Revision

Null is null, != missing. It actually takes space both in the Datastore and Indexes! – Delcine