Loadbalancing web sockets

Question

I have a server which supports web sockets. Browsers connect to my site and each one opens a web socket to www.mydomain.example. That way, my social network app can push messages to the clients.

Traditionally, using just HTTP requests, I would scale up by adding a second server and a load balancer in front of the two web servers.

With web sockets, the connection has to be directly with the web server, not the load balancers, because if a machine has a physical limit of say 64k open ports, and the clients were connecting to the load balancer, then I couldn't support more than 64k concurrent users.

So how do I:

1. get the client to connect directly to the web server (rather than the load balancer) when the page loads? Do I simply load the JavaScript from a node, and the load balancers (or whatever) randomly modifies the URL for the script, every time the page is initially requested?

2. handle a ripple start? The browser will notice that the connection is closed as the web server shuts down. I can write JavaScript code to attempt to reopen the connection, but the node will be gone for a while. So I guess I would have to go back to the load balancer to query the address of the next node to use?

3. I did wonder about the load balancers sending a redirect on the initial request, so that the browser initially requests www.mydomain.example and gets redirected to www34.mydomain.example. That works quite well, until the node goes down - and sites like Facebook don't do that. How do they do it?

Facture

You could load balance at the network layer, as suggested here – Vilma

There are also alternative approaches like DNS based loadbalancing or using a http based orchestration server. I've tried to summarize the up- and downsides of each approach at deepstream.io/blog/load-balancing-websocket-connections – Rica

@Rica Link is dead. :-( – Centrifuge

The link from @Rica looks like it's accessible at deepstreamhub.com/blog/load-balancing-websocket-connections/?io (might need to copy / paste the text to read it though!) – Hal

Answer 1

Put a L3 load-balancer that distributes IP packets based on source-IP-port hash to your WebSocket server farm. Since the L3 balancer maintains no state (using hashed source-IP-port) it will scale to wire speed on low-end hardware (say 10GbE). Since the distribution is deterministic (using hashed source-IP-port), it will work with TCP (and hence WebSocket).

Also note that a 64k hard limit only applies to outgoing TCP/IP for a given (source) IP address. It does not apply to incoming TCP/IP. We have tested Autobahn (a high-performance WebSocket server) with 200k active connections on a 2 core, 4GB RAM VM.

Also note that you can do L7 load-balancing on the HTTP path announced during the initial WebSocket handshake. In that case the load balancer has to maintain state (which source IP-port pair is going to which backend node). It will probably scale to millions of connections nevertheless on decent setup.

Disclaimer: I am original author of Autobahn and work for Tavendo.

Stand

So I would load my javascript library from the load balancer URL and give the load balancer URL when I create the web socket in javascript - you mean it is transparent to the browser? That is cool! – Facture

Yes, there is only 1 URL, and the hostname of the latter should resolve to your load-balancer. The WebSocket backend server have internal IPs (not public), and optionally can run on ports different from the public one also. The only caveat is that you may need to tell the WebSocket servers what their public visible hostname, IP, port is, since conforming WebSocket servers will check that the URL supplied in the HTTP header of the WS handshake fits the hostname/ip/port they are listening on. – Stand

I dont have lot of websocket connections to balance but I have lot of traffic in one or say very few connections. for simplicity say one connection now how can I balance the requests going through one web socket connection? – Hindbrain

When i make more connections 5000+ in java websocket it does not release memory.... is there any solution? – Palatable

Since when 10 GbE was low end hardware in 2012? LOL – Allveta

Can you tell me how will the flow look like? What type of connection will be there between client and load balancer?Is it TCP?Tf yes, then it is as good as having with directly with back server, since there is a limit on number of TCP connections, load balancer cannot handle so many connections and it can possibly handle same amount of connections as a back end node, then what kind of load balancing it did,i did not undestand.Cna you please explain? – Exoenzyme

Along with above doubt, in case of hash based approach, if we want to know like which client is connected to which backend server and which port, then we need to maintain state right,because we can get back end node but not port from hash – Exoenzyme

In the approach I described, there is no TCP state on the LB, as it is a L3/4 LB. It's just IP packets, and the only state to be kept on the LB is about backend server health and the consistent hashing to those. Rgd return traffic: DSR (direct server return) is of course possible in addition. That's a feature of your backend servers. In any case, don't worry about LB performance .. not L4, and not even L7 these days. A modern LB can scale out using L2 features .. it's a non-issue for 99% of all users (unless you need say 10 mio concurrent active websocket connections) – Stand

> When i make more connections 5000+ in java websocket it does not release memory. don't use java;) seriously, if you're stuck on java, I'd recommend looking into Netty. this is a very capable websocket server (and client). the other thing: I assume you are writing a server, and you want to quickly detect broken or even only "stale" (inactive) websocket connections to time out and free mem. you will need websocket ping/pong heartbeating and timers for missed pongs, plus possibly app level activity timers, and then actively close connections from your server code. – Stand

okay so you mean,the communication happens always via load balancer?Client sends data to loadbalancer and then loadbalancer based on sourceip+sourceport sends to the server at back?I did not get this part,can you once tell me exact communication flow ,how it will look like for establishing a TCP connection with backend server?And for the communication after that?What exactly load balancer will do when the TCP connection has to be between client and server only – Exoenzyme

In theory for a L7 load balancer, the 64k client connections per IP should effectively cap the load balancer, as an L7 needs to match every inbound WebSocket with an outgoing WebSocket to the upstream WebSocket servers? – Anthropologist

Answer 2

Note that if your websocket server logic runs on nodejs with socket.io, you can tell socket.io to use a shared redis key/value store for synchronization. This way you don't even have to care about the load balancer, events will propagate among the server instances.

var io = require('socket.io')(3000);
var redis = require('socket.io-redis'); 
io.adapter(redis({ host: 'localhost', port: 6379 }));

See: Socket IO - Using multiple nodes

But at some point I guess redis can become the bottleneck...

Chrystal

Answer 3

You can also achieve layer 7 load balancing with inspection and "routing functionality"

See "How to inspect and load-balance WebSockets traffic using Stingray Traffic Manager, and when necessary, how to manage WebSockets and HTTP traffic that is received on the same IP address and port." https://splash.riverbed.com/docs/DOC-1451

Supply

I had to do some sleuthing to find the info you linked. The wayback machine helped me locate a live copy of that article: community.pulsesecure.net/t5/Pulse-Secure-vADC/… – Seaside