Nginx Reverse Proxy WebSocket Timeout

Asked 29/8, 2018 at 13:1 Answered 19/9, 2019 at 21:51

Solved nginx websocket nginx-reverse-proxy

I'm using java-websocket for my websocket needs, inside a wowza application, and using nginx for ssl, proxying the requests to java.

The problem is that the connection seems to be cut after exactly 1 hour, server-side. The client-side doesn't even know that it was disconnected for quite some time. I don't want to just adjust the timeout on nginx, I want to understand why the connection is being terminated, as the socket is functioning as usual until it isn't.

EDIT: Forgot to post the configuration:

location /websocket/ {
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    include conf.d/proxy_websocket;
    proxy_connect_timeout 1d;
    proxy_send_timeout 1d;
    proxy_read_timeout 1d;
}

And that included config:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass                      http://127.0.0.1:1938/;

Nginx/1.12.2
CentOS Linux release 7.5.1804 (Core)
Java WebSocket 1.3.8 (GitHub)

Safeconduct answered 29/8, 2018 at 13:1 Comment(0)

The timeout could be coming from the client, nginx, or the back-end. When you say that it is being cut "server side" I take that to mean that you have demonstrated that it is not the client. Your nginx configuration looks like it shouldn't timeout for 1 day, so that leaves only the back-end.

Test the back-end directly

My first suggestion is that you try connecting directly to the back-end and confirm that the problem still occurs (taking nginx out of the picture for troubleshooting purposes). Note that you can do this with command line utilities like curl, if using a browser is not practical. Here is an example test command:

time curl --trace-ascii curl-dump.txt -i -N \
  -H "Host: example.com" \
  -H "Connection: Upgrade" \
  -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: BOGUS+KEY+HERE+IS+FINE==" \
  http://127.0.0.1:8080

In my (working) case, running the above example stayed open indefinitely (I stopped with Ctrl-C manually) since neither curl nor my server was implementing a timeout. However, when I changed this to go through nginx as a proxy (with default timeout of 1 minute) as shown below I saw a 504 response from nginx after almost exactly 1 minute.

time curl -i -N --insecure \
  -H "Host: example.com" \
  https://127.0.0.1:443/proxied-path

HTTP/1.1 504 Gateway Time-out
Server: nginx/1.14.2
Date: Thu, 19 Sep 2019 21:37:47 GMT
Content-Type: text/html
Content-Length: 183
Connection: keep-alive

<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

real    1m0.207s
user    0m0.048s
sys 0m0.042s

Other ideas

Someone mentioned trying proxy_ignore_client_abort but that shouldn't make any difference unless the client is closing the connection. Besides, although that might keep the inner connection open I don't think it is able to keep the end-to-end stream intact.

You may want to try proxy_socket_keepalive, though that requires nginx >= 1.15.6.

Finally, there's a note in the WebSocket proxying doc that hints at a good solution:

Alternatively, the proxied server can be configured to periodically send WebSocket ping frames to reset the timeout and check if the connection is still alive.

If you have control over the back-end and want connections to stay open indefinitely, periodically sending "ping" frames to the client (assuming a web browser is used then no change is needed on the client-side as it is implemented as part of the spec) should prevent the connection from being closed due to inactivity (making proxy_read_timeout unnecessary) no matter how long it's open or how many middle-boxes are involved.

Zayas answered 19/9, 2019 at 21:51 Comment(1)

It was a problem with ping like you said. The packge I was using had a implementation for automatically sending pings, but I didn't see it when integrating it, and was sending pings "manually". Once I changed it I did not had any more problems. – Safeconduct 26/9, 2019 at 15:28

Most likely it's because your configuration for the websocket proxy needs tweaking a little, but since you asked:

There are some challenges that a reverse proxy server faces in supporting WebSocket. One is that WebSocket is a hop‑by‑hop protocol, so when a proxy server intercepts an Upgrade request from a client it needs to send its own Upgrade request to the backend server, including the appropriate headers. Also, since WebSocket connections are long lived, as opposed to the typical short‑lived connections used by HTTP, the reverse proxy needs to allow these connections to remain open, rather than closing them because they seem to be idle.

Within your location directive which handles your websocket proxying you need to include the headers, this is the example Nginx give:

location /wsapp/ {
    proxy_pass http://wsbackend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
}

This should now work because:

NGINX supports WebSocket by allowing a tunnel to be set up between a client and a backend server. For NGINX to send the Upgrade request from the client to the backend server, the Upgrade and Connection headers must be set explicitly, as in this example

I'd also recommend you have a look at the Nginx Nchan module which adds websocket functionality directly into Nginx. Works well.

Twibill answered 29/8, 2018 at 16:54 Comment(8)

Sorry, should've posted my current config along with the question. Updated it now. – Safeconduct 29/8, 2018 at 20:52

Have you tried including proxy_ignore_client_abort on;? – Twibill 29/8, 2018 at 21:29

Gonna try it. Will update after some hours with results. Thank you. – Safeconduct 30/8, 2018 at 14:47

Still happening. Several connections closing after exacly one hour :( – Safeconduct 31/8, 2018 at 15:53

I don't think it's a problem with your Nginx setup, or not any you have included here. Are you using SSL anywhere? I had a quick look at some of the code for the websocket server you linked. Noticed a few lines like this: keytool -genkey -keyalg RSA -validity 3650, which appear to generate a certificate with a validity of just over 1 hour, or in Nginx you may need to look at your ssl_session_timeout directives – Twibill 1/9, 2018 at 12:5

I was using the SSL on the java websocket, but changed it to let nginx handle the ssl part, forwarding the connection to a simple non-SSL websocket connection on the same machine. Gonna take a look at session timeout on Nginx, thanks – Safeconduct 2/9, 2018 at 17:28

Tried changing the ssl_session_timeout to 2 hours...but nothing changed. Gonna try to switch back to serving secure websocket from the java lib, to see if the problem is with nginx or the server for some weird reason. – Safeconduct 5/9, 2018 at 13:37

@Twibill -validity 3650 sets the certificate's expiration to 10 years from now (units are days, not seconds, here) – Zayas 19/9, 2019 at 19:49

Test the back-end directly

Other ideas

Recommended topics

Hot tags