Rails app randomly crashes with error "Premature end of script headers"

Question

I am hosting a Ruby on Rails 2.0.2 application on DreamHost. It is on an Apache 2 server, running on top of Phusion Passenger.

The application often returns a 500 error "Rails application failed to start properly", but at random times. It appears to happen when the application is under higher load, though I can't confirm this. It only gets about 2,000 pageviews per day, so I don't think load should really be an issue.

The Apache logs correlate these 500 responses with the error: "Premature end of script headers". Just looking at the logs at 9 AM today, the error appears as often as three or four times a minute. This is clearly unacceptable.

Less frequently, the application hurls a Phusion Passenger page with a stack trace and the error "Broken pipe".

The Rails logs do not list any of these errors.

This happens on both the staging and live sites hosted on DreamHost, but I can't replicate it on a local development server.

So I guess the real question is: Where do I begin to debug this problem?

Kraken

Answer 1

It turns out that I was merely hitting my memory cap on the DreamHost shared server.

I was running several Rails apps under one account, many of them just for testing and prototyping. Rails uses a lot of memory and so I was quickly reaching my allocation. Support told me that "I checked our logs and it's killed one of your ruby processes 2325 times over the last three days". Whoops.

The solution: try not to run Rails in a shared environment, if you can help it. I'm going to switch at least one of my apps to a VPS host soon.

Kraken

Yep this was happening to me on my rails app right after I did some load testing against a PHP app on the same server which caused a lot of apache php.cgi processes to run, thus filling up the memory. Did you happen to find any solutions on reducing memory usage for rails on dreamhost? Thx – Gibbs

There was really no way around it. You could probably host one modestly-popular Rails app on a shared DreamHost server. Regardless, I switched to a VPS hosted with RailsPlayground. I have had no hiccups since then. – Kraken

If you create a new user account on DH, say for every rails app you run, would each account get it's "own" memory? – Ably

I would guess "yes". In addition, your separate accounts would likely be assigned to separate servers (they have hundreds), so they wouldn't run in to memory issues competing with one another. Again, I reiterate that doing Rails in a shared environment will cause you many headaches. – Kraken

Answer 2

Are you running the latest version of passenger (2.2.2 at the time of writing). I had a few errors myself but most (if not all) disappeared after upgrading my passenger installation.

If that's not the answer you can always try to upgrade your rails version to 2.3 and see if the problem persists.

Taoism

I'm checking with DreamHost now to see what version of Passenger they are using. Upgrading Rails to 2.3 might be the next step, though I'm not looking forward to fixing backwards incompatibilities. – Kraken