Load data from S3 into Aurora Serverless using AWS Glue

Question

According to Moving data from S3 -> RDS using AWS Glue

I found that an instance is required to add a connection to a data target. However, my RDS is a serverless, so there is no instance available. Does Glue support this case?

Coth

i did some google search for you, one said, the glue accepts JDBC connections. I haven't used glue personally. – Pharynx

did you find a solution? I tried with JDBC but get "test connection failed". Couldn't find any example on how to configuring VPC, SecurityGroup, Subnet and IAM for this use case. – Halvah

Answer 1

I have tried to connect Aurora MySql Serverless with AWS glue recently, and I failed. And I got a timeout error.

Check that your connection definition references your JDBC database with 
correct URL syntax, username, and password. Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. 
The driver has not received any packets from the server.

I think the reason was Aurora serverless doesn't have any continuously running instances so in the connection URL you cannot give any instances, and that's why Glue cannot connect.

So, you need to make sure that DB instance is running. Only then your JDBC connection works. If your DB runs in a private VPC, you can follow this link: Nat Creation

EDIT: Instead of NAT GW, you can also use the VPC endpoint for S3. Here is a really good blog that explains step by step. Or AWS documentation

Sawtelle

Answer 2

AWS Glue supports the scenario, i.e., it works well to load data from S3 into Aurora Serverless using an AWS Glue job. The engine version I'm currently using is 8.0.mysql_aurora.3.02.0

Note: if you get an error saying Data source rejected establishment of connection, message from server: "Too many connections", you can increase ACUs (currently mine is set to min 4 - max 8 ACUs for your reference), as the maximum number of connections depends on the capacity of ACUs.

Twowheeler

Answer 3

I can use JDBC build connection, There is one thing very important is you should have at least one subnet open ALL TCP port, but you can point the port to the subnet. With the setting, connection test pass, crawler also can create tables.

Knoll