Connection refused to running Google Cloud SQL instance via proxy or from App Engine

Question

I'm very new to Google Cloud and running applications in general. I currently have a Django app running in a Docker container on Google Flexible App Engine that connects to a Google Cloud SQL (PostgreSQL) instance in the same project. The latest version has been running for about 3 days now without issue.

The Problem:

Today I started receiving OperationalError: server closed the connection unexpectedly errors repeatedly from the application. I can run the Cloud SQL Proxy and it starts up normally (Ready for new connections), but if I try to connect with psql, I receive the error:

psql: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

And the proxy reports:

couldn't connect to "<instance_name>:us-central1:<instance_name>": 
dial tcp <ip address>:3307: connect: connection refused

On SSHing into my running flex app instance and running sudo docker logs <cloud proxy container>, the last lines are, similarly:

couldn't connect to "<instance_name>:us-central1:<instance_name>":
dial tcp <ip address>:3307: getsockopt: connection refused

Things I've Tried/Checked

• Restarted cloud sql instance. The instance itself is running fine and I can access it using cloud shell from the console.
• Checked db instance name and ip address - they match.
• Restarted the flex app engine instance. No change as far as I can tell.
• Upgraded my local copy of cloud_sql_proxy to 1.09.
• Checked quotas - I don't seem to have hit any API or simultaneous connection limits.
• I'm able to connect to the sql instance by authorizing my local IP address.
• I'm able to connect to a different (but very similar) Google Cloud SQL instance using the proxy locally so I'm not sure if the proxy is at fault.

Any help at all would be appreciated, at this point I'm out of ideas. Thank you!

Mortise

Answer 1

This could also be an issue if the CloudSQL instance is configured with only a private IP address. Per a small paragraph hidden in the documentation:

The proxy does not provide a new connectivity path; it relies on existing IP connectivity. For example, you cannot use the proxy to connect with an instance using Private IP unless the proxy is using a VPC network that has been configured for private services access.

In this case, the only solution seems to be adding a public IP to the server.

Ambsace

Answer 2

I first restarted the Cloud SQL instance. That did not help. Then, I simply clicked "Stop" for the SQL instance and once it had, clicked "Start" and now it works. This is pretty random and annoying.

In my case, I had upgraded the machine type of the SQL instance earlier in the day and it seems like on doing so, Google Cloud simply "restarts" the instance where as what is needed is "stop" and then "start". This is only a guess.

tl;dr Stop and then Start the Cloud SQL Instance. Don't Restart as "Restart" != "Stop + Start"

Hope it helps others who face this random issue.

Rf

Answer 3

We ended up "fixing" the problem by rolling back to an earlier backup. Google Support noted "the issue started right around the maintenance window for your Cloud SQL instance, so it's possible that a change was made that caused the connection to break".

Mortise