haproxy 504 timeout to apache

Question

Very new to haproxy and loving it, apart from a 504 issue that we're getting. The relevant log output is:

Jun 21 13:52:06 localhost haproxy[1431]: 192.168.0.2:51435 [21/Jun/2017:13:50:26.740] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 13:54:26 localhost haproxy[1431]: 192.168.0.2:51447 [21/Jun/2017:13:52:46.577] www-https~ beFootprints/foorprints 0/0/3/-1/100005 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:15:57 localhost haproxy[1431]: 192.168.0.1:50225 [21/Jun/2017:14:14:17.771] www-https~ beFootprints/foorprints 0/0/2/-1/100004 504 195 - - sH-- 3/3/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:22:26 localhost haproxy[1431]: 192.168.0.1:50258 [21/Jun/2017:14:20:46.608] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1" 

Using the following timeout values in the haproxy.cfg

defaults
        log     global
        mode    http
        option forwardfor
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  100000

Running on Ubuntu 16.04.2 LTS

Any help and comment very much appreciated!

Shorthanded

Thanks for including the log entries, and making this a clean, concise, well-formatted question. You'd be surprised how often that doesn't happen. – Pretender

Answer 1

The problem appears to be with the web server. Check the logs, there, and you should find long-running requests.

Here's how I conclude that.

Note sH-- in your logs. This is the session state at disconnection. It's extremely valuable for troubleshooting. The values are positional and case-sensitive.

s: the server-side timeout expired while waiting for the server to send or receive data.

...so, timeout server fired, while...

H: the proxy was waiting for complete, valid response HEADERS from the server (HTTP only).

The server had not finished (perhaps not even started) returing all the response headers to the proxy, but the connection was established and the request had been sent.

HAProxy returns 504 Gateway Timeout, indicating that the backend did not respond in a timely fashion.

If your backend needs longer than 100 seconds (?!) then you need to increase timeout server. Otherwise, your Apache server seems to have a problem being too slow to respond.

Pretender

Hi Michael, that's what my initial thought was, and the timeouts only happen when the website is authenticating users via ldap. However we do not get a 100s pause when we're logging directly into the webserver, Hit ctrl F5 after 504 error and it goes straight in.... – Shorthanded

There should be something in the application log or the apache access or error logs, or you may need some additonal logging in your application... the cause of the differing behavior needs an explanation, and based on these log entries, the explanation is more likely to be on the app server, not the proxy. – Pretender

@Shorthanded Did you ever solve this issue? I'm on the exact same boat right now. Any help would be greatly appreciated! – Designation

I also hit this issue and it turned out to be a bug in v1.7.10: discourse.haproxy.org/t/… Upgrading to v1.7.11+ fixes the issue. – Contentment

Answer 2

I had a similar issue and found the problem was with how I had configured my backend server section.

backend no_match_backend
  mode http
  balance roundrobin
  option forwardfor
  option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
  server nginx-example 192.168.0.10 check port 80

My problem is that I did not specify the port for the connection. When connecting via HTTP it would work but as I have my SSL terminated on my haproxy. This attempts to connect via 443 to the backends. As the backends cannot / don't correctly communicate. The setup of the SSL session with haproxy and the backend that causes the gateway to time out. I need to force unencrypted communications to the backends.

backend no_match_backend
  mode http
  balance roundrobin
  option forwardfor
  option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
  server nginx-example 192.168.0.10:80 check port 80

The change might be hard to spot server nginx-example 192.168.0.10 check port 80 now has :80 after the ip 192.168.0.10:80

This problem was made more complicated by my backend servers having SSL redirects configured. So all my requests would arrive as HTTP and be redirected to HTTPS. So it was difficult to identify where the problem was. I It looked like https requests were being redirected correctly to the backend servers. I need to disable this redirect on the backend servers and move it forward to haproxy config.

Unnecessarily

That's bizarre, especially because you already state 'port 80' at EOL. – Basra