Why does Passenger keep passenger-error-*.html files in /tmp?

Question

A disk on one of our servers was filling up. Analysis showed that most of the space was wasted in /tmp.

The culprit was the 25,000+ files there that were taking up more than 3 gigs, all of them named after the pattern passenger-error-xxxxxx.html. A quick inspection showed that this was the standard error page Passenger serves when it cannot start an application.

From the message in one of those files, Passenger could not start the application because mysql2 gem was missing and it couldn't connect to the database.

From a rough estimate, it seems as if Passenger kept these files at least for each request which was due to health checks from ELB (request every 30 seconds = 2880reqs/day, Gemfile was fixed after 5 days which should be less than 15,000reqs).

Is it documented anywhere that Passenger keeps these HTML files in /tmp?

Why does it do that? Is something wrong with our config?

Peterson

Answer 1

Passenger author here. Passenger creates such a file every time it fails to spawn a process, for the purpose of allowing administrators to diagnose the problem. Passenger writes an entry to the log file saying something like "Passenger failed to spawn a process, look in /tmp/passenger-error-xxxx.html for details".

If you have 25000+ files, then it just means that Passenger failed to spawn your app 25000+ times. You should definitely go investigate why Passenger failed to spawn your app so often.

Stieglitz

Well, if it's not spawning due to DB misconfiguration the first time and I don't deploy again, it won't spawn the second time. Or third. Or 25,000th. The application server shouldn't fill /tmp over the weekend just because the app was never working. But I regard this as "documented behavior" now :) – Peterson

Passenger has got to log such things somewhere. Writing the entire report to the normal log file doesn't make it easy to read, and that can also fill up the harddisk. You should install a /tmp cleaner if you want to clean up old /tmp files. – Stieglitz

Is there any chance to change the output format and the directory of this file with the details? – Luiseluiza

Passenger respects the TMPDIR environment variable, so make sure you start the web server and Passenger with TMPDIR set to the desired directory. – Stieglitz

With due respect "Making the logs easy to read" doesn't appear to have been a priority Hongli anyway. You made the damn things HTML format which makes them almost impossible to read. What the heck possessed you to put it in a format that can't be read easily on a headless machine?! (ie all webservers , more or less) – Condign

Answer 2

If you need passenger and everything works fine with your app, you can delete old files by creating a cronjob with

crontab -e

and add something like

# m h  dom mon dow   command
23 3 * * * bash find /tmp/passenger-error-* -type f -mtime +1 -exec rm {} \;  > /dev/null 2>&1

Weksler