Using less datastore small operations in appengine

Question

I'm putting together a basic photoalbum on appengine using python 27. I have written the following method to retrieve image details from the datastore matching a particular "adventure". I'm using limits and offsets for pagination, however it is very inefficient. After browsing 5 pages (of 5 photos per page) I've already used 16% of my Datastore Small Operations. Interestingly I've only used 1% of my datastore read operations. How can I make this more efficient for datastore small operations - I'm not sure what these consist of.

def grab_images(adventure, the_offset=0, the_limit = 10):
    logging.info("grab_images")
    the_photos = None
    the_photos = PhotosModel.all().filter("adventure =", adventure)
    total_number_of_photos = the_photos.count()
    all_photos = the_photos.fetch(limit = the_limit, offset = the_offset)
    total_number_of_pages = total_number_of_photos / the_limit
    all_photo_keys = []
    for photo in all_photos:
        all_photo_keys.append(str(photo.blob_key.key()))
    return all_photo_keys, total_number_of_photos, total_number_of_pages

Vue

Answer 1

A few things:

1. You don't need to have count called each time, you can cache it
2. Same goes to the query, why are you querying all the time? cache it also.
3. Cache the pages also, you should not calc the data per page each time.
4. You only need the blob_key but your are loading the entire photo entity, try to model it in a way that you won't need to load all the Photo atributes.

nitpicking: you don't need the_photos = None

Viol

I've made better use of caching and changed the PhotosModel such that blob_key is the actual key (string) rather than the ReferenceProperty. This alone dramatically reduced the number of small datastore queries – Vue

Answer 2

The way you handle paging is inefficient as it goes through every record before the offset to deliver the data. You should consider building the paging mechanisms using the bookmark methods described by Google http://code.google.com/appengine/articles/paging.html.

Using this method you only go through the items you need for each page. I also urge you to cache properly as suggested by Shay, it's both faster and cheaper.

Smashing

Answer 3

You may want to consider moving to the new NDB API. Its use of futures, caches and autobatching may help you a lot. Explicit is better than implicit, but NDB's management of the details makes your code simpler and more readable.

BTW, did you try to use appstats and see how your requests are using the server resources?

Conceivable

I've had a brief look on the NDB API and am a little scared off by the big beta signs everywhere. I'm already using python27 which is not yet exactly production ready and I figure my app is slightly less risky if I only use one beta product at a time... – Vue

Good point. I'm having a fair share of trouble moving an app to 2.7 myself for apparently no good reason (for the trouble, not the migration) :-/ And I agree it's smart to start with 2.7 - it's clear 2.7 is the future for now. – Conceivable