Cloudwatch VPC interface endpoint times out [closed]

Question

I am trying to connect my EC2 host setup inside a private subnet to publish logs to cloud watch. I have setup a VPC interface endpoint for com.amazonaws.us-east-1.logs. Both the interface endpoint and EC2 are in private subnet.

My Security groups

1. Security Group rules for Interface endpoint

   Inbound rule

   Type Protocol Port Range Destination

   All TCP TCP 0 - 65535 sg-OfEC2Server

   Outbound rules - None

2. Security Group rules for EC2

   Inbound rules - None

   Outbound rule

   Type Protocol Port Range Destination

   All TCP TCP 0 - 65535 sg-OfInterfaceEndpoint

The cloudwatch agent keeps timing out, so i sense that there is an issue with my security group rules but none of the trouble shoot steps helped.

Volauvent

Shouldn't the outbound rule be for ` sg-ForEC2Server`? – Trust

updated the question. – Volauvent

The DNS settings for VPC are enabled? – Trust

DNS resolution and DNS hostnames are enabled – Volauvent

I assume the instance has role with permissions to call CW, interfaces policies are also permisive? – Trust

Yep Interface policy is allow everything and I have a role attached to EC2 instance profile that allows the following: "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:CreateLogGroup", "logs:PutLogEvents" – Volauvent

Have you tested manual connection to CW logs, using aws logs cli, just to know that this is not CW agent specific issue? – Trust

Yep this command also times out: aws logs put-log-events --log-group-name vpctest --log-stream-name vpctest --log-events file://data.json --region us-east-1 – Volauvent

When you created the endpoint, Enable DNS name was also enabled? – Trust

I think so, I will create a new endpoint to double-check. Also, do I need to make any changes to the routing table? As per the docs I don't think so – Volauvent

No. Route tables are for Gateway (s3 and ddb) endpoints. – Trust

BTW. How do you connect the the instance? It has no inbout rules? You can't ssh to it. So how did you verify that aws cli doesn't work? – Trust

The new endpoint doesn't work either. I added an inbound rule now I am using a bastion host to ssh into the instance. – Volauvent

I think I know what's happening. I recreated your setup, and got same issue. – Trust

Your outboud rule in EC2 will not allow to contact to DNS server for resolution of dns of the endpoint. Changing output rule to 0.0.0.0/0 fixed the issue. – Trust

Answer 1

I recreated your setup and encountered the same problem when using same security groups as yours.

What I found is that the issue is caused by the following outbound rule:

All TCP TCP 0 - 65535 sg-OfInterfaceEndpoint

This allows EC2 to connect only to the interface endpoint, and nowhere else. Changing it to the CIDR of my VPC (10.1.0.0/16) fixed the problem and I could use the interface endpoint.

I think that the root cause of the timeout was that with the original outbound rule, your (and mine) private instance could not connect to DHCP or DNS servers for the VPC. Subsequently, resolution of DNS name of the logs endpoint was not possible.

Trust

Answer 2

I was having trouble with VPC endpoints timing out when loading ECS images. It wasn't a security group issue but a DNS resolution. I need to add to the endpoint and the private DNS resolved after that:

PrivateDnsEnabled: true  # Enable Private DNS

Prober