Amazon Elastic Load Balancer is not closing the connection to the server

Question

I have an EC2 instance with Apache as a reverse proxy and ffserver as a streaming server. There is an ELB (Classic) in front of the EC2 instance which works as an SSL termination point.

Apache configuration is rather simple:

<Location "/mp3/">
    ProxyPass http://127.0.0.1:8081/ DisableReuse=On KeepAlive=Off
    ProxyPassReverse http://127.0.0.1:8081/
    SetEnv force-proxy-request-1.0.1
    SetEnv proxy-nokeepalive 1
</Location>

ffserver is used to stream live audio over the Internet. In ffserver's settings there is a MaxBandwidth option (default 1000). This setting become a problem when the connections are not closed properly. ffserver starts to respond with 503 server too busy instead of the stream's content.

If I connect to the server directly (no ELB on the road) everything works fine. If I connect over the ELB the connection won't close when I close it on the client's side (e.g. close the browser's tab).

I use the below command to check the current connections:

watch -n 2 'netstat -napt | grep 8081'

All of the connections remain in ESTABLISHED state forever (at least for 30 minutes). The default idle timeout for ELB is 60. It means that someone is receiving the stream from ffserver (connection is active).

EDIT: It looks like changing the Classic Load Balancer to the Application Load Balancer solved the problem. I don't know how to explain this behavior. Looking forward for an answer from AWS community - AWS Forums

Newark

I've heard anecdotal evidence of this, before... as if ELB doesn't sever the connections as it would be expected to when the client disconnects, as long as the server keeps sending data. Just for clarification, is the ELB in TCP mode or HTTP mode? – Herzig

It's HTTP mode. I need X-Forwarded-For header as Apache seems to lack a good support for Proxy Protocol so far. – Newark

Just wondering, does the client have HTTP keep-alive turn on? That would explain why the ELB never timeouts (and would explain also why the ALB solves it). – Poker

You can try to add ping=1 to see if the backend is busy httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass – Lard

Answer 1

As OP shared with his edit, the issue of connections not getting closed can be resolved with a change of load balancer type. This answer focuses on why this change has such an affect?

---

There seems to be an issue within Classic Load Balancer (ELB). I've found the following posts with very similar issues;

• ELB - server keeps sending data to disconnected clients for hours
• Is there a way to lower the LBS KeepAlive?
• AWS ELB servlet client disconnection detection

It seems the issue stems from ELB not being able to detect the client dropping from a connection. Especially when back-end is supplying some sort of data in a periodical manner, e.g. live audio stream, heartbeat, etc.

There doesn't seem to be a way to disable the keep-alive setting of load balancers, yet, somehow, only with ELB this trouble occurs.

I couldn't find exact feature that is creating this behavioural difference between ELB & ALB. I think the reason is either due to;

• Improved Health Checks in ALB, and/or
• an internal structural difference that is not visible to us users, that somehow keeps this issue from happening on ALB

I think the issue resolves when Application Load Balancer (ALB) is used due to said improvements and it being much more flexible.

Check here for more on the differences between ELB, ALB, and NLB

---

ps. AWS support forums are terrible, all the good support & tips are paywalled and stored within PMs between them & their premium customers.

Attlee

Answer 2

I know this is an old thread but I recently had the same issue and the solution was to switch to ALB. Worked correctly after that.

Beetlebrowed