HAProxy closes long living TCP connections ignoring TCP keepalive

Question

I have configured HAProxy (1.5.4, but I tried also 1.5.14) to balance in TCP mode two server exposing AMQP protocol (WSO2 Message Broker) on 5672 port. The clients create and use permanent connection to the AMQP Servers, via HAProxy.

I've changed the client and server TCP keepalive timeout, setting net.ipv4.tcp_keepalive_time=120 (CentOS 7).

In HAProxy I've setted timeout client/server to 200 seconds (>120 seconds of the keepalive packets) and used the option clitcpka.

Then I've started wireshark and sniffed all the tcp traffic: after the last request from the clients, the tcp keepalived packets are sente regularly after 120 seconds, but after 200 seconds after the last request from the clients the connection are closed (thus ignoring the keepalived packet).

Below the configuration:

haproxy.conf

global
    log 127.0.0.1   local3
    maxconn 4096
    user haproxy
    group haproxy
    daemon
    debug

listen messagebroker_balancer 172.19.19.91:5672
    mode tcp
    log global
    retries 3
    timeout connect 5000ms
    option redispatch
    timeout client 200000ms
    timeout server 200000ms
    option tcplog
    option clitcpka 
    balance leastconn
    server s1 172.19.19.79:5672 check inter 5s rise 2 fall 3
    server s2 172.19.19.80:5672 check inter 5s rise 2 fall 3

Droll

Answer 1

TCP keep alive is at the transport layer and is only used to do some traffic on the connection so intermediate systems like packet filters don't loose any states and that the end systems can notice if the connection to the other side broke (maybe because something crashed or a network cable broke).

TCP keep alive has nothing to do with the application level idle timeout which you have set explicitly to 200s:

timeout client 200000ms
timeout server 200000ms

This timeouts gets triggered if the connection is idle, that is if no data get transferred. TCP keep alive does not transport any data, the payload of these packets is empty.

Tito

I'm not an expert of HAProxy, but seems that it operates at TCP level, as described in serverfault.com/questions/589804/… – Droll

@PaoloParlato: the TCP level is large :) HA proxy works at the user space side where you can control keep alive timers but you cannot send or receiveTCP keep alive packets directly. You only can read and write real data which have a size of more than 0 bytes and keep alive packets are empty. And again, the idle timeout in HA proxy cares only about data transfer while TCP keep alive does not transfer any data. – Tito

Answer 2

The timeout client detects a dead client application on a responsive client OS. You can always have an application that occupies a connection but doesn't speak to you. This is bad because the number of connections isn't infinite (maxconn).

Similarly, set timeout server for the backend.

These options were for haproxy talking to application. Now, there is a completely separate check where OS talks to OS (without touching the app or haproxy):

With option clitcpka or option srvtcpka or option tcpka you allow the inactive connection to be detected and killed by the OS, even when haproxy doesn't actively check it. This primarily needs OS settings (Linux).

If no data sent for 110 seconds then immediately send the first keep-alive (KA), don't kill connection yet:

 sysctl net.ipv4.tcp_keepalive_time=110

Wait for 30 seconds after each KA, once they're enabled on this connection:

 sysctl net.ipv4.tcp_keepalive_intvl=30

Allow 3 KAs be unacknowledged, then kill the TCP connection:

 sysctl net.ipv4.tcp_keepalive_probes=3

In this situation OS kills the connection 200 seconds after packets stop coming.

Austro