How to find out why uWSGI kill workers?

Question

i have app on Pyramid. I run it in uWSGI with these config:

[uwsgi]
socket = mysite:8055
master = true
processes = 4
vacuum = true
lazy-apps = true
gevent = 100

And nginx config:

server {
    listen 8050;
    include uwsgi_params;

    location / {
        uwsgi_pass mysite:8055;
    }
}

Usually all fine, but sometimes uWSGI kills workers. And i have no idea why.

I see in uWSGI logs:

DAMN ! worker 2 (pid: 4247) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 4457)

but in the logs there is no Python exceptions.

sometimes i see in uWSGI logs:

invalid request block size: 11484 (max 4096)...skip
[uwsgi-http key: my site:8050 client_addr: 127.0.0.1 client_port: 63367] hr_instance_read(): Connection reset by peer [plugins/http/http.c line 614]

And nginx errors.log:

*13388 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1,
*13955 recv() failed (104: Connection reset by peer) while reading response header from upstream, client:

I think this can be solved by adding buffer-size=32768, but it is unlikely due to this uWSGI kill workers.

Why uwsgi can kill workers? And how can I know the reason? The line "DAMN ! worker 2 (pid: 4247) died, ..." nothing to tells.

Paff

Answer 1

signal 9 means it received a SIGKILL. so something sent a kill to your worker. it's relatively likely that the out-of-memory killer decided to kill your app because it was using too much memory. try watching the workers with a process monitor and see if it uses a lot of memory.

Farlee

"try watching the workers with a process monitor and see if it uses a lot of memory". How do I do this? – Spinescent

Just to add for someone: Use this uwsgitop pypi.python.org/pypi/uwsgitop/0.10 or htop command and filter records to uwsgi and see how much memory is being consumed – Angularity

I'm not sure how it's self-obvious from the uwsgitop command to filter records. When I followed those steps it gave an automatically updating screen in the terminal, although, it did include memory when I included the --memory-report flag. – Infusionism

Answer 2

Try to add harakiri-verbose = true option in the uWSGI config.

Acatalectic

This did not bring anything more for me. – Foretop

Answer 3

I had the same problem, for me changing the uwsgi.ini file, changing the value of the reload-on-rss setting from 2048 to 4048, and harakiri to 600 solved the problem.

Meryl

Answer 4

For me it was that I hadn't filled out app.config["SERVER_NAME"] = "x"

Mcnally