App engine instances spikes

Question

I am using automatic scaling with GAE, running php55. This also happened with php5. Settings:

automatic_scaling:
  min_idle_instances: 0
  max_idle_instances: 2  # default value
  min_pending_latency: 500ms
  max_pending_latency: 7000ms

I am having trouble understanding why these spikes are happening. Typically my application requires no more than 5 instances running at once. Occasionally this will jump to 1,200+ for no apparent reason. The logs surrounding this time show 500 timeout errors for all non-static content.

The only dependency on these pages is a simple database insert. I am using Google Cloud SQL. There are no errors reported in the Cloud SQL logs either.

Any ideas on how to further troubleshoot this? As you can see from the image, this problem is very sporadic but extremely costly.

Korte

Do you have the exact log entry for the 500 error? – Vulgarity

@Vulgarity Every single request during this time frame was a "request timed out after 100 seconds" 500 error. – Korte

Maybe the Cloud SQL request was timing out? Each front end request to GAE must be completed within 60s. You may wish to defer longer running operations to taskqueue: cloud.google.com/appengine/docs/php/taskqueue – Vulgarity

@Vulgarity Thanks for the reply. Unfortunately this unit of work cannot be deferred. I did come across another post that suggested to keep the cloud-sql instance as 'Always On'. Hopefully that will make a difference. Again, thank you. – Korte

Answer 1

This was caused because the database shut down after a short while of inactivity. When traffic came back it took time restart the database server. During this time, app engine thought it needed to create more instances to keep up with the heavy load.

I solved this by leaving my database instance 'always on'.

Korte

like any other resource on the web, the database can go down as well, probably you should handle this scenario when designing your application. Also in app.yaml there's a max_concurrent_requests setting which you can adjust, say if it's 10 and you got 10 process waiting for database IO then appengine will spawn new instance even if the CPU usage of the first instance is zero. I will adjust it to as high as possible. – Stuckey

@LawrenceMok, Be careful with that approach, you can only have a maximum of 12 database connections per app engine instance. Anything after that will fail the DB connection. – Korte