Nginx(as reverse proxy) does not notify gin-gonic(as web server) when connection canceled by client
Asked Answered
F

0

9

In a website which uses gin-gonic as webserver and nginx as a proxy server, clients send their data to the server via gin-gonic exposed APIs, and — in order to send server commands to clients — each one(i.e client) has a connection to the web server which it keeps for a long time(i.e does not expire it for several hours) and fill their response body with the desired command (for bidirectional connection, we can use ws instead of that).

The problem is, we need to know if a client drop/canceled its connection with the server. In order to understand that we have this line case <-c.Request.Context().Done(): in the following code.

func MakeConnection(c *gin.Context) {
// ... code
select {
case notification = <-clientModel.Channels[token]:// we have a command to send to user
    log.Info("Notification")
    err = nil
case <-c.Request.Context().Done(): // connection dropped by client
    log.Info("Cancel")
    err = c.Err()
case <-ctx.Done(): // time elapsed and there was no command to send to client
    log.Info("timeout")
    err = errors.New("timeout exceeded")
}
// ... code
}

Everything works well (if we make such a connection to the server and cancel it after arbitrary time, immediately Cancel displayed in the terminal) until the count of clients increases.

For about 5 clients this work as expected but for more than 10 clients(more load) although nginx logs cancellation in its access log(as error code 499) but webserver (i.e gin-gonic) does not get notified.

nginx configuration

server {
    # SSL configuration
    listen 443 ssl http2 so_keepalive=10:150:1;
    listen [::]:443 ssl http2 so_keepalive=10:150:1;
    include /etc/nginx/snippets/self-signed.conf;
    include /etc/nginx/snippets/ssl-params.conf;

    root /var/www/project;
    autoindex off;

    location / {
        try_files $uri $uri/ @proxy;
    }

    location ~ /v1/v/[0-9a-zA-Z]+/notification {
        proxy_connect_timeout       86460;
        proxy_send_timeout          86460;
        proxy_read_timeout          86460;
        send_timeout                86460;
        proxy_pass                  http://example_host:8002;
    }
}

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    use epoll;
    multi_accept on;
}

http {
    limit_req_zone $binary_remote_addr zone=req_zone:200m rate=10r/s;
    limit_req_zone $request_uri zone=search_limit:100m rate=5r/s;
    limit_req_zone $binary_remote_addr zone=login_limit:100m rate=20r/m;
    vhost_traffic_status_zone;
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    gzip  on;
    gzip_types  text/html text/css text/javascript text/xml text/plain
                image/x-icon image/svg+xml 
                application/rss+xml application/javascript application/x-javascript 
                application/xml application/xhtml+xml 
                application/x-font application/x-font-truetype application/x-font-ttf application/x-font-otf application/x-font-opentype application/vnd.ms-fontobject font/ttf font/otf font/opentype
                font/woff font/ttf
                application/octet-stream;
    gzip_vary on;

    include /etc/nginx/conf.d/*.conf;
}
Facile answered 5/2, 2020 at 14:26 Comment(8)
Hi Bonje, what happens in that case, do you reach the timeout case instead? Couldn't it just be that with more load, requests are getting slower and they are already timed out before even entering this select call?Sprawl
@Sprawl Hi, with more load it timed out. i.e never get the request cancellation and it waits until became timed outFacile
From the information I see then I guess the issue doesn't come from your go code, but probably from a bottleneck in the network or a configuration issue in Nginx. Can you add your Nginx configuration and try increasing the timeout duration to see if that changes anything?Sprawl
I think so, nginx does not send cancellation to web server. Moreover since nginx logs the cancellation I think it is not relevant to timeout duration...Facile
@Sprawl ngixn configuration added. Could you find anything there?Facile
Mh, I don't see anything that seems wrong to me, but I'm no nginx expert to be honest. I'd suggest trying to replace Nginx with another reverse proxy just to check if that fixes it, but that might take significant effort so I guess another option is to wait for someone more knowledgeable than me in Nginx configuration :)Sprawl
can you check for nginx worker_connection value. I came across this article and found it relevant to the above case may be. serverfault.com/questions/787919/…Colman
Possibly duplicated: serverfault.com/questions/240023/… (you don't specify which nginx version you are using, so it is hard to be sure)Rectocele

© 2022 - 2024 — McMap. All rights reserved.