App engine bulk loader download warning "No descending index on __key__, performing serial download"

Question

I'm using the following to download all instances of one of my kinds:

appcfg.py download_data --config_file=bulkloader.yaml --kind=ModelName --filename=ModelName.csv --url=http://appid.appspot.com/remote_api

If the kind has more instances than the batch size, then I get this warning:

No descending index on __key__, performing serial download

I don't have any custom indexes, or any properties with indexes disabled.

Do I 'need' to do something to resolve this warning, or is it just a warning I can safely ignore? Does it effect the speed of the download?

This article on the bulkloader includes the warning message in the sample output, but makes no mention of it.

This post on the app engine group says that I need to create an index. However adding more indexes would slow down writes to my entities - which I'd rather not do as I am going to be writing entities more often than I will be doing bulk data downloads.

Thanks.

Waistband

do you have queries with descending order in your code? – Headlong

@systempuntoout: Yes, but only on one of my kinds (and this warning happens on any kind with > batch size instances), and the order clause is not on the key (it is on a datetime property). – Waistband

Answer 1

As the error describes, without a descending index on __key__ for the model you're downloading, the bulkloader has to download serially. If you add the index as described, it will be able to download in parallel. If you don't, it will work fine, but will be slower to download, as it operates serially.

Note that an additional index has only a small impact on latency, as index rows are written in parallel to the entity write, meaning the write only takes as long as the slowest update.

Ancell

...how do I define a descending index? all I can find are 2-3 year old stuff that produce errors in 2013. :( – Garrulity

Answer 2

I've solved this problem by add this code to index.yaml

kind: books
- properties:
  name: __key__
    - direction: desc
kind: books
- properties:
  name: another_indexes_here

Theophrastus

Answer 3

If your using JAVA and the datastore-indexes.xml file.

Add this (Assuming the name of the kind is "Books") to the datastore-indexes.xml file:

<datastore-index kind="Books" ancestor="false" source="auto"> 
    <property name="__key__" direction="desc"/> 
</datastore-index>

Then redeploy your app. Ensure you check the datastore index tab to see that the __key__ is serving. Then you can try your download again.

Pharisaic