Haproxy Bad Gateway 502

Question

So I am using HAProxy in front of Jetty servlets. The goal at the moment is just proof of concept and load and stress testing once everything's configured. However I have a problem configuring haproxy. I know that it's not a problem with my application cause I have nginx(tengine) running and everything works properly. So it has to be something with the haproxy configuration or just the way haproxy works is not suitable for my needs.

So what my client tries to do is connect to haproxy using two different connections and keep them open:

1. Connect with a chunked streaming mode for upload.
2. Connect with a normal mode and establish a download channel.

Here's how my haproxy.conf file looks like:

global
log /dev/log    local0
log /dev/log    local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
# ca-base /etc/ssl/certs
# crt-base /etc/ssl/private

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
maxconn 2048

defaults
log global
mode    http
option forwardfor
option http-server-close
option  httplog
option  dontlognull
timeout connect 5000
timeout client  50000
timeout server  50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
stats enable
stats uri /stats
stats realm Haproxy\ Statistics
stats auth user:password

frontend www-http
   bind *:80
   reqadd X-Forwarded-Proto:\ http
   default_backend www-backend

frontend www-https
   bind *:443 ssl crt /etc/haproxy/server.pem
   reqadd X-Forwarded-Proto:\ https
   default_backend www-backend

backend www-backend
    redirect scheme https if !{ ssl_fc }
    server www-1 localhost:8080 check maxconn 2048

And here's what my logs say when I try to access port 443:

Sep 17 11:10:18 xxxxx-pc haproxy[15993]: 127.0.0.1:32875 [17/Sep/2014:11:10:18.464] www- https~ www-backend/www-1 0/0/0/-1/1 502 212 - - PH-- 0/0/0/0/0 0/0 "GET /test HTTP/1.1"

Any ideas what the problem might be? An issue with the configuration or ?

Thanks.

Fingered

Answer 1

PH means that haproxy rejected the header from the backend because it was malformed. http://www.haproxy.org/download/1.4/doc/configuration.txt

PH - The proxy blocked the server's response, because it was invalid, incomplete, dangerous (cache control), or matched a security filter. In any case, an HTTP 502 error is sent to the client. One possible cause for this error is an invalid syntax in an HTTP header name containing unauthorized characters. It is also possible but quite rare, that the proxy blocked a chunked-encoding request from the client due to an invalid syntax, before the server responded. In this case, an HTTP 400 error is sent to the client and reported in the logs.

Gusset

Answer 2

I have been searching for a very similar situation where some, not all, requests from a Geoserver backend service was returned as 500 Internal Server Error and the Geoserver has been working flawlessly in the passed. HAProxy logged this as PH-- in its log. Geoserver had nothing in its logs.

For me it turned out that my over eager use of headers for debug purpose was the culprit and after reducing/removing the majority of them the problem stopped.

HAProxy documentation only states that it returns PH-- when the backend HTTP response doesn't comply to standards which is clearly wrong. It also does this when the buffer is overrun or exhausted. Maybe with the difference in HTTP status code 500 instead of 502.

An alternative solution could be to add a larger value to tune.bufsize, tune.h2.header-table-size and/or tune.http.maxhdr in the global section of your config file.

In my case the error disappeared when changing only tune.bufsize value like this (with all the over eager headers still in place):

global
   tune.bufsize 65536    # default 16384

However I kept the default and removed all unnecessary headers instead.

The HAProxy configuration manual states:

HAProxy may emit the following status codes by itself :

 Code  When / reason
                :     (removed for clarity)
  500  when HAProxy encounters an unrecoverable internal error, such as a
       memory allocation failure, which should never happen

But it does, you can argue that it is when used wrong but still. I would have wanted a more elaborated documentation.

Rozanneroze