How do I reload haproxy.cfg on the default Dockerfile?

Question

I'm using the default HAProxy Docker image from https://github.com/dockerfile/haproxy

Unfortunately I can't get it to reload my config properly.

If I run

$ sudo docker exec haprox haproxy -f /etc/haproxy/haproxy.cfg -p '$(</var/run/haproxy.pid)' -st '$(</var/run/haproxy.pid)'

it just dumps out the help file. If I run

$ sudo docker exec haprox 'haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid)'

I get

2014/12/30 00:03:23 docker-exec: failed to exec: exec: "haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid)": stat haproxy -f /etc/haproxy/haproxy.cfg -p $(</var/run/haproxy.pid) -st $(</var/run/haproxy.pid): no such file or directory

Boo. None of those things are what I want. I can run docker exec haprox service haproxy reload - but this ends out spawning several haproxy processes, so when I connect via the unix socket I get one set of information from show stat but I get an entirely different set of information from the http stats page.

I'm trying to set it up so that I can do graceful redeploys of our legacy software, but it does very very bad things with Tomcat sessions, so my only option is to keep existing sessions alive and pinging the same server.

backend legacy
    cookie SERVERID insert indirect preserve
    server A 123.0.0.123:8080 cookie A check
    server B 123.0.0.123:8080 cookie B check

does the trick. I can call up the socket and run set weight legacy/A 0 and it will drain connections from server A.

But (remember that legacy part?) I have to bop my server A/B containers on the head and bring up new ones. I've got my system setup where I'm generating the new config just fine, but when I reload... strange things happen.

As mentioned earlier, it ends out spawning several haproxy processes. I get different information from the stats page and the unix socket. It also appears that the pid file of the process that I'm communicating with in the browser vs. socat are different.

Worst of all, though, is that it will stop http connections with a 503 - and using ab to test will report some dropped connections. This part is not OK.

Old sessions MUST continue to function until the old server goes down/cookies are cleared. It seems like the rest of the Internet is able to do what I'm trying to do... What am I doing wrong here?

Stock

FYI offical HAProxy image is here registry.hub.docker.com/_/haproxy not the one you have linked – Kline

Actually its here now: github.com/docker-library/haproxy – Vesta

Answer 1

if you run ps inside the container as follows you will see the container you have linked runs haproxy as pid 1 which cannot be killed without killing the container and also it is run in foreground so without a pid file. If you want to reload run haproxy in the background in your container and make some other process such as supervisor the primary process.

docker exec -it haproxy ps aux 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
haproxy      1  0.0  0.2  28988  4576 ?        Ss   02:41   0:00 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid

Kline

Answer 2

You can now reload the config:

docker kill -s HUP haproxy_container_name

More info: https://hub.docker.com/_/haproxy

Chiccory

Could you explain how that command is supposed to work? When I run it it just stops the container. – Chrysalid

The -s HUP send a SIGHUP signal to container. This will be reload the daemon. wikipedia: en.wikipedia.org/wiki/SIGHUP : "Daemon programs sometimes use SIGHUP as a signal to restart themselves, the most common reason for this being to re-read a configuration file that has been changed." So when the docker send the SIGHUP signal, the main process will be receiving. If you create an own docker image, and another daemon will be the main process of container, you have to send forward the SIGHUP signal to haproxy daemon. Read the "Reloading config" section from the haproxy documentation. – Chiccory

Thanks! I actually read it and nothing worked as described... but then turned out I had a typo in config :-D now it works. – Chrysalid

I'm always checking the config before reload, example: docker exec haproxy-prod haproxy -c -V -f /usr/local/etc/haproxy/haproxy.cfg && docker kill -s HUP haproxy-prod – Chiccory

Yeah I did it, the config was valid but the last line got ignored because there was no empty line in the end. I don't know whether it was the real issue but after that somehow started to work as expected. – Chrysalid

Looks good but I'm going to be doing it just to reload the ssl certs. Any idea how much downtime and damage restarting like this is gonna do to your web site visitors. Is it worth investing in the hitless reload or the in-memory ssl replacement techniques ? – Myra

Sorry for late answer. I'm using this reload technic when I'm changing the certs. All of the browsers caching the certs while the old cert is valid. So you have to force reload the browser (shift + F5) or try in another browser. – Chiccory