Google App Engine error: NeedIndexError: no matching index found

Question

I'm having trouble with Google's App engine indexes. When running my app via the GoogleAppEngineLauncher, the app is working fine. When deploying the app, I get the following error:

NeedIndexError: no matching index found.
The suggested index for this query is:
- kind: Bar
  ancestor: yes
  properties:
  - name: rating
    direction: desc

The error is generated after this line of code:

 bars = bar_query.fetch(10)

Before the above line of code, it reads:

bar_query = Bar.query(ancestor=guestbook_key(guestbook_name)).order(-Bar.rating)

My index.yaml file contains the exact "suggested" index below # AUTOGENERATED:

- kind: Bar
  ancestor: yes
  properties:
  - name: rating
    direction: desc

Am I maybe missing something? I removed the index.yaml file and deployed the app again (via the command-line) and one less file was uploaded - so the index.yaml file is there.

Everything is working fine locally. I'm working on the latest Mac OSx. The command used for deployment was:

appcfg.py -A app-name --oauth2 update app

The datastore I implemented is loosely based on the guestbook tutorial app.

Any help would be greatly appreciated.

EDIT:

My ndb.Model is defined as follow:

class Bar(ndb.Model):
    content = ndb.StringProperty(indexed=False)
    lat = ndb.FloatProperty(indexed=False)
    lon = ndb.FloatProperty(indexed=False)
    rating = ndb.IntegerProperty(indexed=True)
    url = ndb.TextProperty(indexed=False)

Tumefaction

Answer 1

Check https://appengine.google.com/datastore/indexes to see if this index is present and status set to "serving". It's possible that the index is still being built.

The development environment emulates the production environment. It does not really have indexes in the Datastore sense.

Bunting

Hi @Andrei, thanks, I have been there. It says: "You have not created indexes for this application. Some types of queries require an index to be built. You can manage your indexes in an index.yaml file." On other forums I read that when using the index.yaml it never says "serving". – Tumefaction

I read somewhere the following: "The index that you have put in your datastore-index.xml is a single property index. This means that it is automatically built by Datastore and will not show up in your console." – Tumefaction

You are right - single property indexes do not have to be defined and they won't show up on the index page. I forgot to ask the obvious: is this property indexed? – Bunting

Hi @Andrei thanks for the reply. Yes it is: rating = ndb.IntegerProperty(indexed=True) – Tumefaction

I think it requires a custom index because it is an ancestor query with a descending sort order. In this case this index will appear on the page of indexes. The only explanation for index definitions present in the index.yaml but not on the indexes page is that somehow you are looking at two different applications. Really strange. – Bunting

Thanks @Andrei, I messed around with appcfg.py --help. I found an action 'update indexes'. The console now says pending :D – Tumefaction

Answer 2

Probably a little late now, but running "gcloud app deploy index.yaml" helped since running deploy by itself ignored the index.yaml file.

As others have said, the dashboard at https://appengine.google.com/datastore/indexes will be showing "pending" for a while.

Idell

If you use the appcfg.py or appcfg.sh commands to deploy an application to App Engine, the index configuration file is automatically deployed. If you use the gcloud app deploy command to deploy your application, the index configuration file is not automatically deployed. You must either specifically list index.yaml as an argument to gcloud app deploy or run gcloud datastore create-indexes to deploy the index configuration file. – Antigorite

Answer 3

I stumbled on the same issue and your comments helped me in the right direction. Here's what Google says how to handle this:

According to the Google documentation the story is that using

gcloud app deploy 

the index.yaml file is not uploaded (question is why not?). Anyway, one has to upload this index file manually.

To do so, the documentation gives the following command:

gcloud datastore create-indexes index.yaml

(supposing you execute this from the same directory of the index.yaml file) Once you have done this you can go to the Datastore console and you will see the index has been created. It will then start to be indexed (took some 5 minutes in my case) and once the index is being served you can start your application.

Telephotography

Answer 4

I fixed this issue by moving the index that the error says is missing above the auto generate line in the "index.yaml" file.

In your case the yaml file will look like:

indexes:
- kind: Bar
 ancestor: yes
 properties:
 - name: rating
   direction: desc

# AUTOGENERATED

Then all you have to do is update your app then update the indexes, you update the indexes by running the following command.

appcfg.py [options] update_indexes <directory>

With the directory being the directory relative to your index.yaml file. You should then see that index on your dashboard at https://appengine.google.com/datastore/indexes

The update will initially be "pending" but after the index says "serving" you will be able to make your query.

Anarchism

app.yaml needs to be in the same folder as index.yaml for the above to work. And the command line doesn't like it if index.yaml is in the root folder. And the SDK doesn't like it if app.yaml is in the route folder. Solution: I had to temporarily put both app.yaml and index.yaml in the config folder. – Platelayer

Answer 5

This NeedIndexError can be triggered by different causes, as I arrived here while having a slightly different problem, so I'll try to explain all I was doing wrong in order to show things that can be done:

• I thought I have to had only one index per Kind of entity. That's not true, as long as I found you need to have as many indexes as different queries you will need to make.

• While on development web server indexes are autogenerated and placed below the #AUTOGENERATED line in index.yaml file.

• After modifying indexes I use first gcloud datastore indexes create index.yaml and I wait until indexes are Serving in https://console.cloud.google.com/datastore/indexes?project=your-project.

• I clean unused indexes by executing gcloud datastore indexes cleanup index.yaml be aware that you do not delete indexes that are being used on production. Reference here

• Be aware that if you don't specify direction on your index properties, it will be ASC by default. So if you are trying to make a - sort query it will again rise the error.

Things I think but I have not 100% evidence longer than my particular problem, but I think can help as a kind of brainstorming:

• Indexes are important while querying data, not when uploading.

• Creating manually the #AUTOGENERATED line not seem to be necessary if you are generating indexes manually. Reference here

• As the development server updates indexes below #AUTOGENERATED line while making queries, you can "accidentally" solve your problem by adding this lane. While the real problem is a lack of manually index update using gcloud datastore indexes create index.yaml command. Reference here and here

Vulgate

Answer 6

In my case, I have uploaded the index file manually like below:

gcloud datastore indexes create "C:\Path\of\your\project\index.yaml"

Then you should confirm the update:

Configurations to update:

descriptor:      [C:\Path\of\your\project\index.yaml]
type:            [datastore indexes]
target project:  [project_name]


Do you want to continue (Y/n)?  y

Then you can go to the Datastore console to check if the the index has been created via this link: https://console.cloud.google.com/datastore/indexes

Hepzi