How to properly diagnose a 500 error (Rails, Passenger, Nginx, Postgres)

Question

I'm having a real tough time diagnosing a 500 error from my application running in production. I've had it working before, but after re-deploying via Capastrano I am unable to get it going.

Here are the facts:

1. The server is setup with nginx + passenger, and I'm using PostgreSQL.
2. Static assets are working properly, as in I'm able to access them just fine in a browser.
3. I can access the rails console via RAILS_ENV=production bundle exec rails console and perform Active Record actions (like retrieving data from the db).
4. Within console, I can run app.get("/"), which returns a 500 error as well (after first showing the query that was run to load the model).
5. The production.log file is never written to. I've set permissions 777 on it just for the hell of it. I've also set the log level to :debug with nothing to show for it.
6. The nginx log (which passenger also uses) shows no indication of errors, it just notifies about cache misses.

Because nothing of use is being logged, I have no idea what to do here. I've tried setting full permission on the entire app directory with no help. Restarted the server multiple times, nothing. The database is there and rails can clearly communicate with it. I'm not sure what I did to get it to run the first time around. I just don't know why rails isn't outputting anything to the log.

Hobble

Answer 1

Okay, I figured this out. The app ran fine in development mode, so I knew something production-specific was screwing it up. I went into config/environments/production.rb and changes these settings:

# Full error reports are disabled and caching is turned on
config.consider_all_requests_local       = false # changed from true
config.action_controller.perform_caching = true # changed from false

And then after restarting passenger, rails showed me the error w/ stacktrace. Turns out I was forgetting to precompile the asset pipeline!

Hobble

Answer 2

Things to check

1) Are you sure you are running in production environment? Check to see if any entries are in the development.log file

2) Set up your app to email you when a 500 error occurs with a full stack trace. I use the Exception Notifier gem but there are plenty of other solutions for this.

3) When checking your app in the console are you sure you are starting the console in production mode? It is possible that the app is not starting up at all and you just forgot to set the production param thereby thinking that the app runs fine when it doesn't.

4) Are you getting an nginx 500 error or the Rails 500 error? If nginx then it is likely that your app is not starting at all and highly unlikely that you will get any rails error in your log file. The assets are static files and navigating to them proves nothing other than that they exist.

5) Are you sure you are checking the right folder on the server? Sounds really stupid but capistrano could be deploying the app to a folder that is different to the folder that nginx is looking for for your app so double check both the folder capistrano is deploying to and the folder that nginx is looking for are the same.

Just a suggestion, I would use puma rather than passenger. It's awesome with nginx.

Propagandize

Thanks for all the suggestions! I positive it's running in production mode (I've specified this as a passenger option in nginx config). I'm sure that I was getting the 500 error in Rails, because nginx error log showed no indication of the 500 error. And yes, I double checked the folder nginx was looking at. – Hobble

Answer 3

My problem is passenger's log file (error.log) has nothing. Then it's a rotation log issue. Run

 passenger-config reopen-logs

solved my problem. More.

Schlemiel

Answer 4

Have you tried running in development mode to see if the error reports itself?

Delmardelmer

I tried this, and it first complained about write access to the tmp directory, which I fixed, and then it went through fine. I had it set up using the same database. So it just fails in production mode. – Hobble