Django on GoogleAppEngine: performance howto

Question

I asked this question a few weeks ago. Today I have actually written and released a standard Django application, i.e. a fully-functional relational DB-backed (and consequently fully-functional Django admin) enabled by Google CloudSQL. The only time I had to deviate from doing things the standard Django way was to send email (had to do it the GAE way). My setup is GAE 1.6.4, Python2.7, Django 1.3 using the following in app.yaml:

libraries:
- name: django
  version: "1.3"

However I do need you to suggest clear actionable steps to improve to the response time of the initial request when cold of this Django app. I have a simple webapp2 web site on GAE, which does not hit the DB, and when cold the response time is of 1.56s. The Django one, when cold, hits the DB with 2 queries (two count(*) queries over tables containing less than 300 rows each), and the response time is of 10.73s! Not encouraging for a home page ;)

Things that come to mind are to remove the middleware classes I don't need and other Django-specific optimisations. However tips that improve things also from a GAE standpoint would be really useful.

N.B. I don't want this to become a discussion about the merits of going for Django on GAE. I can mention that my personal Django expertise, and resulting development productivity, did bear considerably in adopting Django as opposed to other frameworks. Moreover with CloudSQL, it's easy to move away from GAE (hopefully not!) as the Django code will work everywhere else with little (or no) modifications. Related discussions about such topic can be found here and here.

Kendricks

What's the response time of the homepage when the application is warm? – Continuant

What have you already profiled? – Tome

@DanielRoseman It averages around the 200ms mark, which is excellent. – Kendricks

Well, a good start would be to identify and remove those count queries. Also, have you enabled warmup requests? – Night

@NickJohnson Didn't enable warmup requests - I've read about the warmup requests but I'm still at an initial stage and I'm focusing on single-instance improvements right now. As far as I understood warmup requests work when starting a new instance correct? – Kendricks

@JosvicZammit Correct - it allows the system to spin up a new instance in the background, so your users don't see the long startup time. – Night

Answer 1

I don't have a full answer but I'm contributing since I'd like to find a solution as well. I'm currently using a running cron job (I actually need the cron job, so it's not only for keeping my app alive).

I've seen it discussed in one of the GAE/Python/Django related mailing lists that just the time required for loading up all the Django files is significant when compared to webapp, so removing django components that you don't use from deployment should improve your startup time as well. I've been able to shave off about 3 seconds by removing certain parts of the contrib folder. I exclude them in my app.yaml.

My startup time is still around 6 seconds (full app, Django-nonrel, HRD). It used to be more like 4 when my app was simpler.

My suspicion is that Django verifies all its models on startup, and that processing time is significant. If you have time with experimenting with an app with absolutely 0 models, I'd be curious if it made any impact.

I'm also curious as to whether your two initial queries make any significant impact.

Milksop

Warmup requests are designed for this - there's no need for a cronjob to do it. – Night

@Milksop Issue must be related to Django being heavy and querying CloudSQL... as with WebApp2 when cold it responds always with the ~2s mark. I'm doing some tweaks/improvements will let you know the outcome. – Kendricks

@NickJohnson I focused on django-only optimizations, by removing the django-admin from being enabled on the production instance. Based on whether the application is running in production or locally I set the Django setting variable DEBUG, and now I also add to the MIDDLEWARE_CLASSES and INSTALLED_APPS if I'm running locally. I also removed ~600Kb of Django admin static media (they are not available automatically in production). These Django-focuse tweaks improved my response time when cold from the ~10s above to ~5s. – Kendricks

@NickJohnson Next step was to remove the queries that hit CloudSQL. I removed my 2 count(*) queries. And now the response time is always ~3s. – Kendricks

Josvic, I suspect most of the time is just disk load time. My Django folder alone is 30MB, that takes time to load (webapp2 is much smaller). So not only do you want to remove extra stuff from INSTALLED_APPS, you don't want them to be read from disk at all. Try excluding as much of django/contrib as you can, using the "skip_files:" directive in app.yaml. @NickJohnson: I need a cron job to clear out timed-out requests (specific to my app) anyways, so it was a bonus. – Milksop

@Milksop In my case files that get uploaded on appcfg update total ~900K. I use the app.yaml libraries statement above to get Django.. maybe that still incurs loading time as you mention. Thing is if I make data in production readonly then I can simply use an in-memory DB (sqlite won't block in read-only scenarios) rather than CloudSQL... – Kendricks

Ah thanks for pointing that out. I forgot you were using the included Django. – Milksop

Answer 2

When there is no instance running, for example after version upgrade or when there is no request for 15 min, then a request will trigger loading of an instance which takes about 10s. So what you are seeing is normal.

So if your app is idle for periods longer then 15min you will see this behavior. One workaround is to have a cron job ping your instance every 10 min (though I believe google does not like that).

Update:

You can avoid this by enabling billing then in GAE admin under "Application settings" set minimum Idle Instances setting to 1. Note: the min setting is not available on free apps, only max.

Overabundance

So it's the same issue as this ... but how come this 10s warm up does not happen with webapp2 apps? – Kendricks

Don't know. I see it on my java test apps. – Overabundance