Accessing AWS RDS using IAM Authentication and Spring JDBC (DataSource and JdbcTemplate)
Asked Answered
A

3

18

I am not able to figure out how to implement this. Any help and/or pointers will be greatly appreciated.

Currently, my Java/Spring application backend is deployed on EC2 and accessing MySQL on RDS successfully using the regular Spring JDBC setup. That is, storing database info in application.properties and configuring DataSource and JdbcTemplate in @Configuration class. Everything works fine.

Now, I need to access MySQL on RDS securely. RDS instance has IAM Authentication enabled. I have also successfully created IAM role and applied inline policy. Then, following the AWS RDS documentation and Java example on this link, I am able to access the database from a standalone Java class successfully using Authentication Token and the user I created instead of regular db username and password. This standalone Java class is dealing with "Connection" object directly.

The place I am stuck is how I translate this to Spring JDBC configuration. That is, setting up DataSource and JdbcTemplate beans for this in my @Configuration class.

What would be a correct/right approach to implement this?

----- EDIT - Start -----

I am trying to implement this as a library that can be used for multiple projects. That is, it will be used as a JAR and declared as a dependency in a project's POM file. This library is going to include configurable AWS Services like this RDS access using general DB username and password, RDS access using IAM Authentication, KMS (CMK/data keys) for data encryption, etc.

Idea is to use this library on any web/app server depending on the project.

Hope this clarifies my need more.

----- EDIT - End -----

DataSource internally has getConnection() so I can basically create my own DataSource implementation to achieve what I want. But is this a good approach?

Something like:

public class MyDataSource implements DataSource {
    @Override
    public Connection getConnection() throws SQLException {
        Connection conn = null;
        // get a connection using IAM Authentication Token for accessing AWS RDS, etc. as in the AWS docs
        return conn;
    }

    @Override
    public Connection getConnection(String username, String password) throws SQLException {
        return getConnection();
    }

    //other methods
} 
Arizona answered 14/6, 2018 at 0:14 Comment(1)
Anybody? I can't be the first one trying to do this...Arizona
R
13

You can use the following snippet as a replacement for the default connection-pool provided by SpringBoot/Tomcat. It will refresh the token password every 10 minutes, since the token is valid for 15 minutes. Also, it assumes the region can be extracted from the DNS hostname. If this is not the case, you'll need to specify the region to use.

public class RdsIamAuthDataSource extends org.apache.tomcat.jdbc.pool.DataSource {

private static final Logger LOG = LoggerFactory.getLogger(RdsIamAuthDataSource.class);

/**
 * The Java KeyStore (JKS) file that contains the Amazon root CAs
 */
public static final String RDS_CACERTS = "/rds-cacerts";
/**
 * Password for the ca-certs file.
 */
public static final String PASSWORD = "changeit";
public static final int DEFAULT_PORT = 3306;

@Override
public ConnectionPool createPool() throws SQLException {
    return pool != null ? pool : createPoolImpl();
}

protected synchronized ConnectionPool createPoolImpl() throws SQLException {
    return pool = new RdsIamAuthConnectionPool(poolProperties);
}

public static class RdsIamAuthConnectionPool extends ConnectionPool implements Runnable {

    private RdsIamAuthTokenGenerator rdsIamAuthTokenGenerator;
    private String host;
    private String region;
    private int port;
    private String username;
    private Thread tokenThread;

    public RdsIamAuthConnectionPool(PoolConfiguration prop) throws SQLException {
        super(prop);
    }

    @Override
    protected void init(PoolConfiguration prop) throws SQLException {
        try {
            URI uri = new URI(prop.getUrl().substring(5));
            this.host = uri.getHost();
            this.port = uri.getPort();
            if (this.port < 0) {
                this.port = DEFAULT_PORT;
            }
            this.region = StringUtils.split(this.host,'.')[2]; // extract region from rds hostname
            this.username = prop.getUsername();
            this.rdsIamAuthTokenGenerator = RdsIamAuthTokenGenerator.builder().credentials(new DefaultAWSCredentialsProviderChain()).region(this.region).build();
            updatePassword(prop);
            final Properties props = prop.getDbProperties();
            props.setProperty("useSSL","true");
            props.setProperty("requireSSL","true");
            props.setProperty("trustCertificateKeyStoreUrl",getClass().getResource(RDS_CACERTS).toString());
            props.setProperty("trustCertificateKeyStorePassword", PASSWORD);
            super.init(prop);
            this.tokenThread = new Thread(this, "RdsIamAuthDataSourceTokenThread");
            this.tokenThread.setDaemon(true);
            this.tokenThread.start();
        } catch (URISyntaxException e) {
            throw new RuntimeException(e.getMessage());
        }
    }

    @Override
    public void run() {
        try {
            while (this.tokenThread != null) {
                Thread.sleep(10 * 60 * 1000); // wait for 10 minutes, then recreate the token
                updatePassword(getPoolProperties());
            }
        } catch (InterruptedException e) {
            LOG.debug("Background token thread interrupted");
        }
    }

    @Override
    protected void close(boolean force) {
        super.close(force);
        Thread t = tokenThread;
        tokenThread = null;
        if (t != null) {
            t.interrupt();
        }
    }

    private void updatePassword(PoolConfiguration props) {
        String token = rdsIamAuthTokenGenerator.getAuthToken(GetIamAuthTokenRequest.builder().hostname(host).port(port).userName(this.username).build());
        LOG.debug("Updated IAM token for connection pool");
        props.setPassword(token);
    }
}
}

Please note that you'll need to import Amazon's root/intermediate certificates to establish a trusted connection. The example code above assumes that the certificates have been imported into a file called 'rds-cacert' and is available on the classpath. Alternatively, you can also import them into the JVM 'cacerts' file.

To use this data-source, you can use the following properties for Spring:

datasource:
  url: jdbc:mysql://dbhost.xyz123abc.us-east-1.rds.amazonaws.com/dbname
  username: iam_app_user
  driver-class-name: com.mysql.cj.jdbc.Driver
  type: com.mydomain.jdbc.RdsIamAuthDataSource

Using Spring Java config:

@Bean public DataSource dataSource() { 
    PoolConfiguration props = new PoolProperties(); 
    props.setUrl("jdbc:mysql://dbname.abc123xyz.us-east-1.rds.amazonaws.com/dbschema"); 
    props.setUsername("iam_dbuser_app"); 
    props.setDriverClassName("com.mysql.jdbc.Driver"); 
    return new RdsIamAuthDataSource(props); 
}

UPDATE: When using MySQL, you can also decide to use the MariaDB JDBC driver, which has builtin support for IAM authentication:

spring:
  datasource:
    host: dbhost.cluster-xxx.eu-west-1.rds.amazonaws.com
    url: jdbc:mariadb:aurora//${spring.datasource.host}/db?user=xxx&credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
    type: org.mariadb.jdbc.MariaDbPoolDataSource

The above requires MariaDB and AWS SDK libraries, and needs the CA-bundle in the classpath

Rhebarhee answered 25/6, 2018 at 12:29 Comment(13)
blagerweij - this is awesome!! I did not think of going the route of connection pooling because of the limitation of RDS connections. This probably would have been my next question if and when I came at that stage! Thanks for your great help and pointing in this direction with a proper implementation, very much appreciated! I need to ask one more favor from you, would you be kind enough to give a sample code utilizing this? Thanking you in advance.Arizona
If you're using Spring Boot, all you need to do is add a dependency with a JdbcTemplate or DataSource. (e.g. @Autowired JdbcTemplate jdbcTemplate)Rhebarhee
Hi blagerweij - I have updated my original question with more info and the correct need I have (I think). Can you please have a look and provide some pointers? Your help is greatly appreciated! Thank you.Arizona
Please note, this is not to say that your solution above is not needed. In fact, it is perfect and a great way to solve the limitations RDS connection has. But I am not able to utilize it for my purpose of creating a library.Arizona
The solution presented above uses the DefaultAWSCredentialsProviderChain, which means it supports all available mechanisms (environment vars, profile, Java properties, etc). We are using the above snippet in a shared library for many projects that require AWS specific services. Spring Boot is used for configuring the bean properties, the only thing you will need is to specify the class-name of the DataSource (type: com.mydomain.jdbc.RdsIamAuthDataSource)Rhebarhee
blagerweij - first thing, I would like to give a big thank you for keeping your patience in reading my replies and giving the answers to my queries! I really very highly appreciate it! ; Now, I did try it in the same exact way as you said...specifying the class-name of DataSource as RdsIamAuthDataSource. The only difference is I am not using Spring Boot, mine is a regular Spring/Spring JDBC application. Would that make any difference? Thank you!Arizona
Yes, that makes all the difference. Spring Boot has support for autoconfiguration, which automatically configures a DataSource. If you use Spring with XML config, you need to write custom config: <bean id="dataSource" class="com.mydomain.jdbc.RdsIamAuthDataSource" destroy-method="close"> <property name="url" value="xxx" /> <property name="driverClassName" value="com.mysql.cj.jdbc.Driver" /> <property name="username" value="iam_user" /> </bean> . Although I recommend using Boot, it makes your life easier.Rhebarhee
Got it, thanks for confirming. Actually, it's an existing project with Spring JDBC dependency already given in POM. And configuration is done using pure Java (i.e. using @Configuration annotation on ApplicationConfig class which has a @Bean declared as public DataSource dataSource() method and in this method a DriverManagerDataSource is being created and the properties like URL, DriverClassName, UserName and Password are being set. If I change this method to RdsIamAuthDataSource I don't get the methods to set all those properties. Any suggestions how I can go about it? Thank you.Arizona
You can probably use something like this: (cannot get the format right, sorry): ` @Bean public DataSource dataSource() { PoolConfiguration props = new PoolProperties(); props.setUrl("jdbc:mysql://dbname.abc123xyz.us-east-1.rds.amazonaws.com/dbschema"); props.setUsername("iam_dbuser_app"); props.setDriverClassName("com.mysql.jdbc.Driver"); return new RdsIamAuthDataSource(props); }`Rhebarhee
Here's a sample project: wetransfer.com/downloads/…Rhebarhee
This is AWSome! :) I am going to try it today. blagerweij - I really can't thank you enough for this! I really very highly appreciate your continued patience and help! Best Regards!Arizona
Hi blagerweij - I finally got to try it now but running into an error. Thought of running it by you while I try to troubleshoot it at my end. Please do let me know if you know the cause or give me some pointers. I am getting com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: java.io.ByteArrayInputStream@61dbedc2 does not appear to be a valid URL. when a call to super.init(prop) is made from within our init() method. The DB URL is good so not sure what the error is exactly about. Thanks for your much needed help!Arizona
Partial stacktrace where I think it's failing: ...at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:310) ~[tomcat-jdbc-8.5.11.jar:na] at org.....jdbc.pool.PooledConnection.connect(PooledConnection.java:203) at org.....jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:732) at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:664) at org.....jdbc.pool.ConnectionPool.init(ConnectionPool.java:479) at com.prj.service.util.RdsIamAuthDataSource$RdsConnectionPool.init(RdsIamAuthDataSource.java:82)...Arizona
U
10

Updated for 2023!

This can now be accomplished even more easily using the AWS JDBC Driver for MySQL support for IAM Database Authentication. I tested out this configuration with Spring Boot 3 & spring-boot-starter-jdbc.

Add the following dependencies:

runtimeOnly 'software.aws.rds:aws-mysql-jdbc:1.1.6'
runtimeOnly 'software.amazon.awssdk:rds:2.20.57'

Add the following to application.yml:

spring:
  datasource:
    jdbc-url: jdbc:mysql:aws://yourdbcluster-xxxx.cluster-xxxx.your-region.rds.amazonaws.com:3306/yourdb?useAwsIam=true
    username: iam_username
    #password: dont-need-this
    driver-class-name: software.aws.rds.jdbc.mysql.Driver

And it should just work!

The driver uses the AWS default credentials provider chain so make sure you have credentials with policy allowing IAM DB access available wherever you are running your app. If you are also using failover support, make sure to read all the driver documentation on github. Hope this update helps someone else!


*This is an older method using MariaDB driver - see updates above!

I know this is an older question, but after a some searching I found a pretty easy way you can now do this using the MariaDB driver. In version 2.5 they added an AWS IAM credential plugin to the driver. It will handle generating, caching and refreshing the token automatically.

I've tested using Spring Boot 2.3 with the default HikariCP connection pool and it is working fine for me with these settings:

spring.datasource.url=jdbc:mariadb://host/db?credentialType=AWS-IAM&useSsl&serverSslCert=classpath:rds-combined-ca-bundle.pem
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.username=iam_username
#spring.datasource.password=dont-need-this
spring.datasource.hikari.maxLifetime=600000

Download rds-combined-ca-bundle.pem and put it in src/main/resources so you can connect via SSL.
You will need these dependencies on the classpath as well:

runtime 'org.mariadb.jdbc:mariadb-java-client'
runtime 'com.amazonaws:aws-java-sdk-rds:1.11.880'

The driver uses the standard DefaultAWSCredentialsProviderChain so make sure you have credentials with policy allowing IAM DB access available wherever you are running your app.

Hope this helps someone else - most examples I found online involved custom code, background threads, etc - but using the new driver feature is much easier!

Ulphi answered 20/10, 2020 at 19:14 Comment(2)
But it still requires the AWS_SECRET_ACCESS_KEY to be set...Cobos
I stand corrected. I tested it and the MariaDB driver will pick up the instance profile of your EC2 instance. This makes this solution really easy to implement. Give it a try!Cobos
T
0

There is a library that can make this easy. Effectively you just override the getPassword() method in the HikariDataSource. You use STS to assume the role and send a "password" for that role.

<dependency>
  <groupId>io.volcanolabs</groupId>
  <artifactId>rds-iam-hikari-datasource</artifactId>
  <version>3.0.0</version>
</dependency> 
Tripod answered 18/6, 2022 at 19:0 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.