Docker Swarm HAProxy Not Load Balancing w/ Overlay Networking

Question

I have spent the past few day working on creating a docker swarm on Digtital Ocean. Note: I don't want to use -link to communicate with the other apps/containers becasue they are technically considered deprecated and don't work well with docker swarm (i.e. I can't add more app instances to the load balancer without re composing the entire swarm)

I am using one server as a kv-store server running console according to this guide. Becasue i'm on Digital Ocean, i'm using private networking on DO so the machines can communicate with each other.

I then create a hive master and slave, and start the overlay network, which is running on all machines. Here is my docker-compose.yml

proxy:
    image: tutum/haproxy 
    ports:
        - "1936:1936"
        - "80:80"

web:
    image: tutum/hello-world
    expose:
        - "80"

So when I do this it creates the 2 containers. HAProxy is running because I can access the stats at port 1936 at http://<ip-address>:1936, however, when I try to go to the web server/load balancer at port 80 I get connection refused. I everything seems to be connected though, when I run docker-compose ps:

       Name                      Command               State                                 Ports
--------------------------------------------------------------------------------------------------------------------------------
splashcloud_proxy_1   python /haproxy/main.py          Up      104.236.109.58:1936->1936/tcp, 443/tcp, 104.236.109.58:80->80/tcp
splashcloud_web_1     /bin/sh -c php-fpm -d vari ...   Up      80/tcp

The only thing I can think of is that it's not linking to the web container, but i'm not sure how to troubleshoot this.

I'd appreciate any help on this.

Sowder

This is just a link to a related discussion for the dockerized nginx-reverseproxy. I did not create an answer, because it's not about haproxy, but it might be an alternative. – Semifinal

Thanks @schmunk. This may be useful. I can't use Nginx in this case because I need the queuing capabilities of HAProxy - only the paid version of Ngjnx supports queuing. – Sowder

i'm trying to find in the docker documentation the statement that links are considered deprecated, but i cannot find anything, could you please point out where did you read that? – Schindler

I have seen it written since docker 1.9, as the networking is now auto setup and before networking was not so automated. i.e. anything in a compose file would auto get a network on the machine names to hosts file. I can't find a reference at the moment. – Groomsman

Found a reference for you (and @SalvadorJuanMartinez) In the Multi Host Networking announcment, blog.docker.com/2015/11/docker-multi-host-networking-ga you make a named network before you start the container. – Groomsman

@SalvadorJuanMartinez I read that links should be considered deprecated in a Github issue from a Docker contributor. I can find the specific issue if need be, but basically he said to use networking because it's more dynamic and useful. – Sowder

thanks for the link @Groomsman and Zach Rusell, i took a look at it, and yea, networking seems to have nice advantages, i'll reconsider my practices for future projects. – Schindler

Answer 1

you cannot use the tutum haproxy version here unfortunately. This image is specifically tailored around links. You do need some scripted way of passing the web server ip to haproxy I fear.

But this isn't all that hard :) I would suggest you start from this example: First setup the docker-compose.yml => lets use two nodes, just so you can make sure what you're doing makes sense and actually load balances along the way :)

proxy:
    build: ./haproxy/
    ports:
        - "1936:1936"
        - "80:80"
web1:
    container_name: web1
    image: tutum/hello-world
    expose:
        - "80"
web2:
    container_name: web2
    image: tutum/hello-world
    expose:
        - "80"

Now with haproxy you need to setup your own Dockerfile according to the official images documentation: https://hub.docker.com/_/haproxy/

I did this in the haproxy subfolder using the suggested file:

FROM haproxy
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

then for the haproxy config file haproxy.cfg I tested this:

global
    stats socket /var/run/haproxy.stat mode 660 level admin
    stats timeout 30s
    user root
    group root

defaults
    mode    http
    timeout connect 5000
    timeout client  50000
    timeout server  50000

frontend localnodes
    bind *:80
    mode http
    default_backend nodes

backend nodes
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server web01 172.17.0.2:80
    server web02 172.17.0.3:80

listen stats 
    bind *:1936
    mode http
    stats enable
    stats uri /
    stats hide-version
    stats auth someuser:password

Obviously the IPs here will only work in the default setup I'm fully aware of this :) You need to do something about those 2 lines:

server web01 172.17.0.2:80
server web02 172.17.0.3:80

I think you're in luck here working with Digital Ocean :) As far as I understand you do have private IP addresses at your disposal with DO under which you are planning to run the swarm nodes. I suggest to simply put those node IPs instead of my example IPs and run your web servers on them and you're good :)

Ratsbane

Thanks for your response. It's funny, the reason I switched to using Tutum's image was because I was having issues with the official image - figures. I am taking advantage of DO's private networking, so hopefully i'll be able to take advantage of that to add the IP addresses. I am seeing this repo which pay prove useful for what i'm looking for as well. I'll let you know how it goes once I try it! – Sowder

@ZachRussell since you already found my little haproxy demo I'd only like to add another link to the official haproxy blog where you might find inspiration. I would be interested in your final result and if you like, you might even contribute to the GitHub repo with additional details or corrections. Thanks! – Convulse

Thanks @Convulse i'm still stuck with what i'm trying to accomplish, but I feel like i'm getting closer! – Sowder

@ZachRussell if I can assist anyhow, just leave a note - as already mentioned I'm also curious for a real life scenario. – Convulse

Answer 2

Add links to your docker-compose.yml

proxy:
    image: tutum/haproxy 
    ports:
        - "1936:1936"
        - "80:80"
    links:
        - web
web:
    image: tutum/hello-world
    expose:
        - "80"

Kitty