Can I have sticky sessions with HAProxy and socket.io with authentication?

Question

I have several instances of socket.io with authentication running under HAProxy and I need to force that the authentication request and the socket connection go to the same instance. I've set up HAProxy based on this answer to a SO question with some modifications as so:

global
    maxconn     4096 # Total Max Connections. This is dependent on ulimit
    nbproc      2

defaults
    mode        http

frontend all 0.0.0.0:80
    timeout client 86400000
    default_backend www_backend
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket hdr_beg(Host) -i ws

    use_backend socket_backend if is_websocket

backend www_backend
    balance url_param sessionId
    option forwardfor # This sets X-Forwarded-For
    timeout server 30000
    timeout connect 4000
    server server1 localhost:8081 weight 1 maxconn 1024 check
    server server2 localhost:8082 weight 1 maxconn 1024 check
    server server3 localhost:8083 weight 1 maxconn 1024 check

backend socket_backend
    balance url_param sessionId
    option forwardfor # This sets X-Forwarded-For
    timeout queue 5000
    timeout server 86400000
    timeout connect 86400000
    server server1 localhost:8081 weight 1 maxconn 1024 check
    server server2 localhost:8082 weight 1 maxconn 1024 check
    server server3 localhost:8083 weight 1 maxconn 1024 check

I've tried url_param (where sessionId is a querystring parameter passed in both the authentication call and the websocket connection) and source as the balance options but it seems as if HAProxy only allows these options for HTTP connections and so ignores them for the actual websocket connection. The result is that sometimes the auth request and the socket connection end up in different servers, which is unacceptable for our application.

Is there some way to have this desired behavior?

Superintendency

Did you ever figure out the best way to do this? I was looking to do the same thing. Thanks. – Benelux

I ended up using IP-hash balancing. All requests from the same IP go to the same server – Superintendency

O ok, but does that mean that a websocket being broadcasted from a user on server A does not reach any of the users on server B? – Benelux

You need to use a separate database to store the sessions. Socket.io has support for a redis database to store the socket session information across different nodes, just point the different socket.io servers to the same redis db and it should work. – Superintendency

Answer 1

I use cookie based balancing in this way:

backend socketio
    mode http
    cookie SIO insert
    server sock1 127.0.0.1:8001 cookie 001
    server sock2 127.0.0.1:8002 cookie 002

Actuate

For the cookie method, is there a way to set the cookie timeout period? – Carrissa

Answer 2

To balance TCP connection, you may have some success with a stickiness table using the stick_match or stick on commands and explicitly setting tcp mode.

Here is an example:

# forward SMTP users to the same server they just used for POP in the
# last 30 minutes
backend pop
    mode tcp
    balance roundrobin
    stick store-request src
    stick-table type ip size 200k expire 30m
    server s1 192.168.1.1:110
    server s2 192.168.1.1:110

backend smtp
    mode tcp
    balance roundrobin
    stick match src table pop
    server s1 192.168.1.1:25
    server s2 192.168.1.1:25

Full documentation available here.

Harrietharriett

Answer 3

For websocket connections balance using roundrobin. Since its a Bidirectional socket (over TCP) stickyness is maintained by default. For other transports using source balancing algorithm is the best bet. (You can use cookie based persistence but socket.io doesn't send a JSESSIONID or the like back to the proxy server. You can try sockjs if you want cookie based persistence.)

Example:

#do the same for other transports. [Note in 0.6.x resource was mounted at path: /socket.io/[transport]
acl is_JSONPolling path_beg /socket.io/1/jsonp-polling
use_backend non_websocket if is_JSONPolling


backend non_websocket
  balance source
  #rest same as the one for websocket backend

Blackguard

The problem is that the authentication request and the websocket connection may end up in different servers. Is there any way of having them always go to the same server? Would you know if any other load balancer provides this? – Superintendency

It doesn't matter which server is hit provided your auth store is separated from the app. Hopefully you are not using memory as your store for auth sessions. Use something like Redis to store the auth key. Have a look at my answer here for handling auth: https://mcmap.net/q/167528/-socket-io-authentication/… – Blackguard

Answer 4

You're using HTTP so insert a cookie for persistence - that's definitely the best route. That'll stick it to the first server they went to unless it's down.

You can also configure whether it should redispatch it if it is down etc.

Catnip

Answer 5

In my case I needed to authenticate the user using an Authorization: Bearer ${ACCESS_TOKEN}, for that kind of authentication you can use stick_tables.

The following configuration sets the balancer-host port 80 to track and stick whose connections where the Authorization header is the same:

frontend :80
   mode http
   bind :80
   default_backend http_servers

backend http_servers
    balance roundrobin
    stick-table type string len 600 size 1000m expire 5m
    stick on req.hdr(Authorization)
    server server81 192.168.1.1:81 weight 1 maxconn 512 check
    server server82 192.168.1.1:82 weight 1 maxconn 512 check
    server server83 192.168.1.2:83 weight 1 maxconn 512 check

Uniocular