Google app engine excessive datastore small operations

Question

My site has about 50 users and I am getting excessive small datastore operations. I am aggressively memcaching, dont have that many records and still I get millions of small datastore operations. Appstats says the cost is 0 yet the real cost is not 0. I basically know where the small datastore operations might occur.

Key only operations: I do this but I memcache it until the data is not changed. Plus most of my key only operation have limit=100 (this is max) so to get 12m operations I would need to make 120000 calls (I am assuming fetching 1 key is 1 small operation). As I get about 60-70 visits a day that seems a bit excessive. I just cant figure out what is causing that many operations. Appstats is giving me no clue.

This is the dashboard.

This is the appstats.

Lovett

Do you use the remote API at all? Is appstats enabled for your entire application? – Medalist

Yes appstats is enabled according to this developers.google.com/appengine/docs/python/tools/… Second figure is RPC stats. – Lovett

Well it could be possible that a section of your code isn't wrapped using the appstats wrapper. You'd have to ensure that every endpoint in your app uses this. For instance, if you use mapreduce it does not use app stats. All actions via the Remote API also does not use app stats. App Stats only store the most recent 1000 entries in memcache, so its possible its not even giving you an accurate overview of your app (especially with all the recent issues with short-lived memcache entries) – Medalist

Answer 1

Are you using lots of counts? Seems like this can be a problem that causes excessive datastore small operations.

I don't have your code, but this answer has some suggestions for optimizing your code when experiencing this problem.

Also, take a look at a similar question - Google app engine excessive small datastore operations for similar answers

Hygroscope

Thanks for the answer, I am not doing too many counts and I am caching wherever possible. In any case 12m operations for 50 users seems bit out of the park. Some funky stuff must be happening with my code, I wish there was a way to do some analysis in development side. My code is over 35KLOC so cant feasibly post here. – Lovett

You are talking about this when you say appstats, righT? – Irish

Yes that is exactly what I mean look at the screenshot. – Lovett

Answer 2

I notice this old question isn't yet solved, so based on your info, here is another potential cause.

Running my GAE SDK on an very fresh public Azure VM instance (xxx.cloudapp.net), I've noticed a lot of bot traffic coming in trying to find a common open source CMS or cart's admin page. I believe this is due to the bots either utilizing AXFR requests or bruteforce detection of subdomains.

Make sure that you are blocking any unwanted bot traffic and not serving them a dynamic page, hitting your datastore more.

The same condition can also be caused by a rogue AJAX request looping on every page that those 50 users request.

Takahashi