At least one security group must open all ingress ports. AWS Glue connecting to RDS

Question

I am still starting out with AWS Glue and I am trying to connect it to my publicly accessible MySql database hosted on RDS Aurora to get its data.

So I start by creating a crawler and in the data store I create a new connection as in the screenshot below:

I go through the rest and eventually try to run the crawler but I get the following error: At least one security group must open all ingress ports.To limit traffic, the source security group in your inbound rule can be restricted to the same security group

I am not sure what I need to change in the security group attached to the RDS but here's what I have right now for the inbound rules:

You'll notice that I have a self-referencing rule in there that's pointing to the same security group.

The outbound rules are going to all traffic.

Any idea what I might be doing wrong?

Impossibly

Answer 1

The inbound rule (Glue Connection security group) is set to allow TCP Port 0 to allow traffic. Instead, it should allow ALL traffic. Edit your rules, and where there's a dropdown that says "Custom TCP Rule", and change it to "All TCP".

The documentation explains how to setup the security group

Science

Thanks, I am still getting an error though but I think that's more related to the output of the crawler: VPC S3 endpoint validation failed for SubnetId: subnet-1944ab40. VPC: vpc-c8605bad. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-1944ab40 in Vpc vpc-c8605bad – Impossibly

Yeah, that's a different problem, you may want to create a separate question. I believe it's relating to the need to set up what's called an "S3 Gateway Endpoint" in your VPC / subnet. – Science

Did the second issue ever get resolved? Experiencing the same issue – Iormina

Answer 2

You need to set a new rule in the security group that is attached to your DB instances where you define:

• Type: All TCP
• Protocol: TCP
• Range: 0 - 65535
• Source: Custom sg-(the id of this/self security group)
• Description: whatever you want

Bigham

Answer 3

To solve the second error mentioned above in the comments (VPC S3 endpoint validation failed for SubnetId: subnet-1944ab40. VPC: vpc-c8605bad. Reason: Could not find S3 endpoint or NAT gateway for subnetId: subnet-1944ab40 in Vpc vpc-c8605bad) you have to create an Amazon VPC Endpoints for Amazon S3. https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html

Unkindly

Answer 4

I found @David I. Rock solution to be working but has the inconvenience to stop connections via SQL Clients.

On top of that I also added the inbound rule:

• Type: MYSQL / Aurora
• Protocol: TCP (automatically generated)
• Port Range: 3306 (automatically generated)
• Source: My IP (or adapt to your requirements)

Tyranny