Apache load balance tomcat websocket

Question

I am currently developing a websocket application, which is deployed on a Tomcat server. Because of the huge number of users I would like to distribute the workload to multiple Tomcat instances. I decided to use Apache for load balancing.

Now I have a problem with the implementation of Apache load balancing and sticky session for websockets requests. This is my Apache configuration:

ProxyRequests off
SSLProxyEngine on
RewriteEngine On

<Proxy balancer://http-localhost/>
    BalancerMember  https://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  https://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

<Proxy balancer://ws-localhost/>
    BalancerMember  wss://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  wss://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

RewriteCond       %{HTTP:Upgrade} =websocket
RewriteRule       /jddd/(.*)      balancer://ws-localhost/$1 [P,L]
ProxyPassReverse  /jddd/          balancer://ws-localhost/
RewriteCond       %{HTTP:Upgrade} !=websocket
RewriteRule       /jddd/(.*)      balancer://http-localhost/$1 [P,L]
ProxyPassReverse  /jddd/          balancer://http-localhost/

The first https request is balanced to Port 8443. The upgraded wss request is also forwarded to 8443.

The second https request contains the sessionID of the first request: https://...&sid=C28C13EEEC525D203F8CA4E827605E0B.jvm1

As I can see in the Apache log file, this sessionID is evaluated for stickySession:

...Found value C28C13EEEC525D203F8CA4E827605E0B.jvm1 for stickysession sid

...Found route jvm1

...balancer://http-localhost: worker (htttps://mcsgest1.desy.de:8443/Whiteboard/) rewritten to htttps://mcsgest1.desy.de:8443/Whiteboard//?file=octocenter.xml&address=///&sid=C28C13EEEC525D203F8CA4E827605E0B.jvm1

The second https request is still on port 8443, but after upgrading to websocket protocol, the ws-balancer doesn't evaluate the sessionID and rewrites to 8444:

...balancer://ws-localhost: worker (wss://mcsgest1.desy.de:8444/Whiteboard/) rewritten to wss://mcsgest1.desy.de:8444/Whiteboard//whiteboardendpoint

What do I have to change in the Apache config to enable stickysession also for wss protocol? Do I really need two balancers (http and ws) to get websockets balanced?

Ruche

Answer 1

You don't need separate balancer for websokets, because initial http request already has http cookie and belong to right instance.

You just need detect connection upgrade and manually route request depending of sticky part of the cookie

Make sure you load proxy module for websokets - mod_proxy_wstunnel

for example

SSLProxyEngine on
RewriteEngine On

<Proxy balancer://http-localhost/>
    BalancerMember  https://mcsgest1.desy.de:8443/Whiteboard/ route=jvm1 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    BalancerMember  https://mcsgest1.desy.de:8444/Whiteboard/ route=jvm2 keepalive=On smax=1 connectiontimeout=10 retry=600 timeout=900 ttl=900
    ProxySet lbmethod=byrequests
    ProxySet stickysession=JSESSIONID|sid scolonpathdelim=On
</Proxy>

RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm1 [NC]
RewriteRule .* wss://mcsgest1.desy.de:8443%{REQUEST_URI} [P,L]

RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm2 [NC]
RewriteRule .* wss://mcsgest1.desy.de:8444%{REQUEST_URI} [P,L]

RewriteRule  /jddd/(.*)  balancer://http-localhost$1  [P,L]

ProxyPreserveHost On
ProxyRequests Off

ProxyPass /jddd/  balancer://http-localhost
ProxyPassReverse /jddd/ balancer://http-localhost

Explanation:

# if header upgrade = WebSocket
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]  
# and header connection contains Upgrade (header may be like this: connection=keep-alive, upgrade)
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] 
# and header cookie contains JSESSIONID or sid, ending with sticky part - .jvm1 in that case
RewriteCond %{HTTP_COOKIE} ^.*(JSESSIONID|sid)=([^=]*)\.jvm1 [NC] 
#than we route request to application server via mod_proxy (P flag) and end rewrite rule check
RewriteRule .* wss://mcsgest1.desy.de:8443%{REQUEST_URI} [P] 

Refusal

My http requests are on / and my websocket requests are on /push. Will this work for both load balancing http requests and websocket requests? – Aubrey

Did you need to repeat the upgrade section? I mean the: RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC] RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC] – Aubrey

I follow your instructions to set up, however, from Chrome console, the jvm# not added to the session id. is there any special config need to do? – Nonbeliever

@MaoshengWang you should set up jvmRoute in tomcat config, for example <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1"> – Refusal

my backend, not tomcat, it seems I need to find another way. – Nonbeliever