AWS ECS ALB Error (Request timed out)

Question

I am trying to learn/use AWS ECS but keep getting

service has reached a steady state.

Followed by:

service (instance i-05873e2a55ecba2f6) (port 32768) is unhealthy in target-group due to (reason Request timed out)

I'm not really sure which info you need to help, but I was using this load balancer across EC2 instances before, but I am replacing those EC2 instances with ones launched through ECS and now I am running into this error.

My cluster is in my default VPC and I am including all 3 subnets (East zone). The security group is my load balancer security group which allows all traffic on ports 40 and 443. I have tried changing security group so that it allows anyone on any port but that doesn't work.

My host port in my task definition is 0 and my container port is 3000 which is what I exposed in Dockerfile.

The healthcheck is just on the target port at path "/"

Lucullus

what is instance security group? is the port 32768 open at instance level? – Mislay

Yeah, I have tried that – Lucullus

are you able to connect to app from the host container? curl <ip>:3000 – Mislay

can you confirm if the healthcheck port is 32768 or the one on which the service is running ? – Shipyard

@SudharsanSivasankaran If I SSH into the EC2 instance and then try to curl <ip>:3000, I get a connection timed out – Lucullus

@kintuparantu I don't think it is a healthcheck issue, but the healthcheck is checking on the traffic port (which should be the assigned port) – Lucullus

So, you need to fix the timeout problem first and make <ip>:3000 work – Mislay

@SudharsanSivasankaran The timeout is fixed but now I am running into a healthcheck error "Health checks failed with these codes: [302]" – Lucullus

302 is for url redirecting, pls see my answer for fixing #49390010 – Mislay

Answer 1

Remember to check the outbound rule of your ALB security group. Target group health check actually issues the request from ALB. So if your ALB is not allowed to talk to your target, it will also fail.

Deadpan

Thanks! I was just about to lose the plot on this one. – Melodic

Answer 2

This answer summarize a checklist of points to verify when debugging this kind of error:

• be the case, there is no route Path /healthcheck in the backend service
• The status code from /healthcheck is not 200
• Might be the case that target port is invalid, configure it correctly, if an application running on port 8080 or 3000 it should be 3000 or 8080
• The security group is not allowing traffic on the target group
• Application is not running in the container

Urine

If you are using Terraform to deploy some restricted SG, don't forget to add the self attribute so the port will be allowed on your own SG (which will be the one used for ELB taget) : registry.terraform.io/providers/hashicorp/aws/latest/docs/… – Liston

Answer 3

My problem was the same. Check the inbound rule of the security group of the ALB, there should be something like this. All traffic / All / All / "sg-xxxxxxxxxxxx" –.

sg-xxxxxxxxxxxx this should be the security group of your application load balancer.

Dwight

Answer 4

My problem was that even after fixing the instances' security group to properly allow ingress from the load balancer, it still wasn't working. The issue turned out to be that an instance was retaining its incorrect security group from before, since it had termination protection from a setting. It would then get automatically picked up by the target group, despite the new security group i had defined for it. After force terminating it everything was fixed.

Vyborg