Is it best to query by keys_only=True then get_multi or just full query?

Question

I am using NDB with python 2.7 with threadsafe mode turned on.

I understand that querying for entities with NDB does not use local cache or memcache but goes straight to the datastore unlike getting by key name. (The rest of the question might be redundant if this premise is not correct.)

Therefore would a good paradigm be to only query with keys_only=True and then do a get_multi to obtain the full entities?

The benefits would be that keys_only=True queries are much faster than keys_only=False, get_multi could potentially just hit memcache & by calling get_multi your entities are now saved in memcache in case you need to do the query again.

The drawbacks are that you now have an RPC query call + get_multi call and I think there is a limit to how may entities you can call in one get_multi therefore your effective query size might be limited.

What do you think? Should we only ever query using keys_only=True then perform get_multi? Are there certain minimum and maximum query size limits that make this technique not as effective as just doing a query that returns the full entities?

Insufferable

Consider your application data access pattern, if a significant portion of your query results are unlikely to be in cache, or unlikely to be used again for a while then you will always have a high cache miss rate. As to direct performance comparisons between keys only query and get_multi vs a normal query - why don't you run some tests on your anticipated use pattern. – Private

Answer 1

This has been extensively researched. See http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=118

Penultimate

Thanks for the link. To summarise there is benefit of doing a keys_only query then a get_multi with the entity keys returned if you know that the majority of your entities are in memcache already. – Insufferable