Spring Boot application in Google App Engine can't connect to Cloud SQL

Question

I'm not sure what else I'm missing here, but my application is failing on start with:

Message    : The connection attempt failed.

    at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:60) ~[flyway-core-5.2.4.jar!/:na]
    at org.flywaydb.core.internal.database.DatabaseFactory.createDatabase(DatabaseFactory.java:72) ~[flyway-core-5.2.4.jar!/:na]
    at org.flywaydb.core.Flyway.execute(Flyway.java:1670) ~[flyway-core-5.2.4.jar!/:na]
    at org.flywaydb.core.Flyway.migrate(Flyway.java:1356) ~[flyway-core-5.2.4.jar!/:na]
    at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:65) ~[spring-boot-autoconfigure-2.1.7.RELEASE.jar!/:2.1.7.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837) ~[spring-beans-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774) ~[spring-beans-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
    ... 26 common frames omitted
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.Driver.makeConnection(Driver.java:458) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.Driver.connect(Driver.java:260) ~[postgresql-42.2.6.jar!/:42.2.6]
    at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:136) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-3.2.0.jar!/:na]
    at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:56) ~[flyway-core-5.2.4.jar!/:na]
    ... 32 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na]
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[na:na]
    at java.base/java.net.Socket.connect(Socket.java:591) ~[na:na]
    at org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]
    ... 44 common frames omitted    

My app.yaml is

runtime: java11
manual_scaling:
  instances: 1
beta_settings:
  cloud_sql_instances: recruiters-wtf:europe-west2:recruiters-wtf=tcp:5432

while my application.properties contains:

spring.datasource.url=jdbc:postgresql://172.17.0.1:5432/${DB_NAME:recruiters_wtf}
spring.datasource.username=${DB_USER}
spring.datasource.password=${DB_PASS}

What else am I missing? The database is being successfully used by a Kubernetes deployment of the same application.

Following kurtisvg's answer, I change the connection string to:

spring.datasource.url=jdbc:postgresql://google/recruiters_wtf?cloudSqlInstance=recruiters-wtf:europe-west2:recruiters-wtf=tcp:5432&socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=${DB_USER}&password=${DB_PASS}

I already had postgres-socket-factory installed. That change the error to:

Message    : The connection attempt failed.

    at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:60) ~[flyway-core-5.2.4.jar!/:na]
    at org.flywaydb.core.internal.database.DatabaseFactory.createDatabase(DatabaseFactory.java:72) ~[flyway-core-5.2.4.jar!/:na]
    at org.flywaydb.core.Flyway.execute(Flyway.java:1670) ~[flyway-core-5.2.4.jar!/:na]
    at org.flywaydb.core.Flyway.migrate(Flyway.java:1356) ~[flyway-core-5.2.4.jar!/:na]
    at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:65) ~[spring-boot-autoconfigure-2.1.7.RELEASE.jar!/:2.1.7.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837) ~[spring-beans-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774) ~[spring-beans-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
    ... 26 common frames omitted
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.Driver.makeConnection(Driver.java:458) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.Driver.connect(Driver.java:260) ~[postgresql-42.2.6.jar!/:42.2.6]
    at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:136) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:369) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:198) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:467) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:541) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-3.2.0.jar!/:na]
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-3.2.0.jar!/:na]
    at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:56) ~[flyway-core-5.2.4.jar!/:na]
    ... 32 common frames omitted
Caused by: java.io.IOException: No such file or directory
    at jnr.unixsocket.UnixSocketChannel.doConnect(UnixSocketChannel.java:127) ~[jnr-unixsocket-0.23.jar!/:na]
    at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:136) ~[jnr-unixsocket-0.23.jar!/:na]
    at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:68) ~[jnr-unixsocket-0.23.jar!/:na]
    at com.google.cloud.sql.core.CoreSocketFactory.connect(CoreSocketFactory.java:180) ~[jdbc-socket-factory-core-1.0.15.jar!/:na]
    at com.google.cloud.sql.postgres.SocketFactory.createSocket(SocketFactory.java:71) ~[postgres-socket-factory-1.0.15.jar!/:na]
    at org.postgresql.core.PGStream.<init>(PGStream.java:67) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]
    ... 44 common frames omitted

Megganmeggi

Have you tried removing the beta_settings entry in your app.yaml file? It's not defined for GAE Standard for Java11 – Belting

Answer 1

As documented here:

App Engine standard enviroments do not support connecting to the Cloud SQL instance using TCP. Your code should not try to access the instance using an IP address (such as 127.0.0.1 or 172.17.0.1) unless you have configured Serverless VPC Access.

First, you'll need to remove the beta_settings entry in your app.yaml file since:

• it is intended for App Engine Flexible (and actually instruct the platform to spin up a sidecar container running the CloudSQL proxy, which you don't need with Socket Factory for JDBC)
• the syntax you used with =tcp:5432 would actually instructs the CloudSQL proxy to connect via TCP port.

So your Base JDBC url should look like jdbc:postgresql://google/recruiters_wtf

Your full datasource url would be (note the removed =tcp:5432 part):

jdbc:postgresql://google/recruiters_wtf?cloudSqlInstance=recruiters-wtf:europe-west2:recruiters-wtf&socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=${DB_USER}&password=${DB_PASS}

Finally, note that, as mentioned in the link I provided above:

The PostgreSQL standard requires the Unix socket to have a .s.PGSQL.5432 suffix in the socket path. Some libraries apply this suffix automatically, but others require you to specify the socket path as follows: /cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432.

Belting

I'm getting this error Unable to obtain connection from database: FATAL: password authentication failed for user "${DB_USER}" but that's progress! I would have sworn I tested this combinations of settings, but clearly I didn't. – Megganmeggi

I'm going to explore from here, but if you have any recommendations, they are welcome. – Megganmeggi

you need to ensure the service account used has the correct role/permissions (Cloud SQL Client for example), although the connection seems successful so it seems to be a permission issue to the database itself. Have you tried hard-coding the username and pwd as a test? – Belting

Hard-coding the username and password works and the app is up and running. It is still not clear to me how the username and password should be correctly and securely handled. – Megganmeggi

As in... should they somehow be injected by App Engine based on some configuration or should I use KMS – Megganmeggi

You can set them as environment variables in the app.yaml file. – Belting

If you prefer to avoid hard-coding them in the app.yaml file to be able to commit the file in a source repository you may want to have look at these SO answers here and here – Belting

Those two answers seem to be offloading the work to Cloud Build, which I'm not using and I'm not sure I will though. – Megganmeggi

Answer 2

I'm seeing that you are using DB_NAME:recruiters_wtf, shoudn't that be DB_NAME?

Attempt to use: spring.cloud.gcp.sql.database-name if not.

Also please make sure to deploy with beta

Let me know.

Distraction

DB_NAME:recruiters_wtf resolves to the environment variable DB_NAME and if not to recruiters_wtf. – Megganmeggi

Answer 3

It's likely that one of two things is occurring that allow your k8s cluster to connect: 1. You are using a Private IP 2. You are using a Public IP, and added your k8s cluster to the whitelist

If you are using a private IP to connect, you'll have to configure Serverless VPC Access to allow App Engine to connect.

If you are using a Public IP, you won't be able to add App Engine to your whitelist because there is no guarantee what IP it connects from. Instead, I suggest using the Cloud SQL JDBC Socket Factory to authenticate your connections instead.

Bienne

Thank you for the answer. I am indeed using a public IP. I had the Cloud SQL JDBC Socket Factory installed, but maybe improperly configured. Correcting the configuration changed the error. – Megganmeggi

Answer 4

Ensure that the service account for your service has one of the following IAM roles:

Cloud SQL Client (preferred) Cloud SQL Editor Cloud SQL Admin Or, you can manually assign the following IAM permissions:

cloudsql.instances.connect cloudsql.instances.get

Cracked