gevent-socketio nginx uwsgi not working together on development server

Question

I am running a django project that uses gevent-socketio.

For some reason on my development server, all my websockets requests are returning a 101 pending message at which socketio will start cycling through all the other protocols which result in a pending status.

The error in my uwsgi logs is:

2013/05/23 16:09:08 [error] 14485#0: *85 upstream timed out (110: Connection timed out) while reading upstream, client: x.x.x.x, server: dev.proj.co, request: "GET /socket.io/1/xhr-polling/116404981619?t=1369325348489 HTTP/1.1", upstream: "http://127.0.0.1:4042/socket.io/1/xhr-polling/116404981619?t=1369325348489", host: "dev.proj.co", referrer: "http://dev.proj.co/map/bycon/"

Locally, I do not have this problem. I start the server using python run.py

run.py on my local environment

#!/usr/bin/env python
import os
import sys

from gevent import monkey
monkey.patch_all()

import django.core.handlers.wsgi
from socketio.server import SocketIOServer

import os

PORT = 8000

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
application = django.core.handlers.wsgi.WSGIHandler()
PROJECT_DIR            = os.path.realpath(os.path.dirname(__file__))
sys.path.insert(0, PROJECT_DIR)
sys.path.insert(0, os.path.join(PROJECT_DIR, "chat"))

if __name__ == '__main__':
    SocketIOServer(('', PORT), application, resource="socket.io").serve_forever()

On my development server, where the bug is occurring, I have the following settings:

nginx.conf

worker_processes  auto;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    gzip  on;
    upstream django {
        server 127.0.0.1:4042;
    }


    server {
        listen       80;
        server_name dev.proj.co;
        charset utf-8;

        access_log  /var/log/nginx/myproj_dev.access.log;
        error_log  /var/log/nginx/myproj_dev.error.log;

        location /media/ {
            alias /var/www/dev/myproj/releases/myproj_public/media/;
            error_page 404 = /404;
            expires 30d;
        }

        location /static/ {
            alias /var/www/dev/myproj/releases/myproj_public/static/;
            error_page 404 = /404;
            expires 30d;
        }
        location / {
            proxy_pass http://127.0.0.1:4042;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }
}

uwsgi_dev.ini

[uwsgi]

if-env = PROJECT_HOME
print = Your path is %(_)/current
chdir = %(_)/current
touch-reload = %(_)/current/myproj/uwsgi_dev.ini
daemonize = %(_)/myproj_uwsgi/myproj.log
endif =

if-env = VIRTUAL_ENV
print = Your virtualenv is %(_)
virtualenv = %(_)
endif = 

gevent = 100
processes = 4
module = myproj.wsgi_dev
env = DJANGO_SETTINGS_MODULE=myproj.settings.dev
master = True
vacuum = True
max-requests = 5000
logdate = True

# newrelic requirements
enable-threads = True
single-interpreter = True

wsgi_dev.py

import os

from gevent import monkey
monkey.patch_all()

from socketio.server import SocketIOServer
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproj.settings")
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

PORT = 4042
SocketIOServer(("127.0.0.1", PORT), application, \
        resource="socket.io").serve_forever()

Other info:

I'm using uwsgi 1.9.6, gevent-websocket 0.3.6, greenlet 0.4.0 and the dev version of gevent(-e git://github.com/surfly/gevent.git@e2f51c1358e88b60e45d1daf8ee263da64066576#egg=gevent-dev) and gevent-socketetio (-e git://github.com/abourget/gevent-socketio.git@aeece7038b0052ddf6b4228857e4d7a67a6242f2#egg=gevent_socketio-dev)

[root@li476-12 ~]# nginx -v
nginx version: nginx/1.4.1

Unbosom

Can you maybe try running it with one process and see if the problem persists? I'm also having troubles with gevent-socketio with multiple workers using gunicorn, not sure if it's related github.com/abourget/gevent-socketio/issues/132?source=cc. Additionally using uwsgi with gevent seems to be very buggy... github.com/abourget/gevent-socketio/issues/81?source=cc – Leeland

@BernhardVallant Thanks for the comment. The gunicorn link was quite helpful. I don't have much time to debug this now but when I do, I'll try to remember to update this post. – Unbosom

@Unbosom I know this is an old question, but I'm wondering if you ever solved what was giving you the error? I'm encountering a similar issue currently. – Cheeks

Answer 1

For websockets support you need latest release of nginx (1.4.x). If you use older releases you will not be able to start the websocket channel

Villalpando

Answer 2

The SocketIOServer will never run as you are running the WSGI django file as a module so __name__ is not __main__

In addition to this the logic is wrong as SocketIOServer will takeover uWSGI blocking yorur django app.

I suggest you to run SocketIOServer from a separate script using --attach-daemon of uWSGI

(and obviously put uWSGi on a different port)

Villalpando

Thanks for responding @roberto. I think the phrasing of my question has confused you. I've amended to make it clearer. I'm only using the run.py file (as opposed to ./manage.py runserver) locally and it works ok. The problem is the settings on my development server. – Unbosom

Answer 3

Disable threads in UWSGI.

You have problem "Global Interpretator Lock"

enable-threads = True <- i think problem here, try False

Enrique