How to manage passenger stale workers that causes downtime

Question

I have a rails app running on ubuntu 14.04 machine and it is served by Nginx and passenger. There are times when the app goes down giving the following error

[ agents/LoggingAgent/Main.cpp:338 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ agents/LoggingAgent/Main.cpp:400 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ ServerKit/Server.h:453 ]: [LoggerAdminServer] Shutdown finished
[ agents/LoggingAgent/Main.cpp:425 ]: PassengerAgent logger shutdown finished
[ ServerKit/Server.h:453 ]: [ServerThr.1] Shutdown finished
[ ServerKit/Server.h:453 ]: [ServerThr.2] Shutdown finished
[ ServerKit/Server.h:453 ]: [AdminServer] Shutdown finished
[ agents/HelperAgent/Main.cpp:724 ]: Disconnecting long-running connections for process 19431, application /public#default
[ agents/HelperAgent/Main.cpp:724 ]: Disconnecting long-running connections for process 19440, application /public#default
[ agents/Watchdog/Main.cpp:1255 ]: Starting PassengerAgent watchdog...
[ agents/HelperAgent/Main.cpp:883 ]: Starting PassengerAgent server...
[ agents/HelperAgent/Main.cpp:232 ]: PassengerAgent server running in multi-application mode.
[ agents/HelperAgent/Main.cpp:637 ]: PassengerAgent server online, PID 19637
[ agents/LoggingAgent/Main.cpp:431 ]: Starting PassengerAgent logger...
[ agents/LoggingAgent/Main.cpp:312 ]: PassengerAgent logger online, PID 19645
[ agents/HelperAgent/Main.cpp:868 ]: PassengerAgent server shutdown finished

And the app remains down unless the Nginx is being restarted. This is probably due to the stale workers which are not serving anymore and eating up the memory (correct me if I am wrong).

I have set the max pool size of passenger to 6 as my app instance takes 150-200 Mb of memory with a 2GB RAM.

I was thinking to write a script which regularly check for the stale workers and if found restart the Nginx or do a touch tmp/restart.txt. I also want to know how to check for the stale workers or if the passenger is not working as expected.

Another solution I thought is to regularly restart Nginx or do a touch tmp/restart.txt after a regular interval of time, say one hour.

I have also read the passenger documentation about the three ways of restarting the app

• passenger-config restart-app

• restart.txt

• Restarting Nginx

The first two cases won't drop any request while restarting the app but may have zero down time while restarting the app as the restarting of app may take some time depending on the app.

I want to know which would be a good way or is there any other way to get rid of this problem.

Servility

Answer 1

Firstly, you should really check passenger configuration options. Secondly, you should really figure out what actually hangs your passenger worker. For that there's a passenger log parameter (which you can monitor), as it does not seem that you're quite certain what is going on in those processes. :)

Now regarding the first issue at hand - your wish to force worker restarts - if that's actually due to process hanging while doing nothing: you can combine

passenger_min_instances
passenger_max_instances

with

passenger_pool_idle_time / passenger_max_requests

Read a bit about these commands on the given link, it goes into greater detail.

Countermine