unicorn timeout handling

Question

I'm wondering what is the best way to track unicorn timeouts from my app.

There are some bits of the app which are slow and they currently timeout quietly. I could increase the timeout but that is shoving the problem under the carpet. Ideally, I would like to get an airbrake notification or something of that effect. I'm not looking for a performance guide, just a way to efficiently and reliably get informed about timeouts.

How have other people handled timeouts from their rails app using unicorn? scan the nginx error log? plug in a handler in unicorn config? handler in nginx config?

[Ubuntu 12.04+nginx+unicorn+rails 3.2+ruby 1.9.3]

Jessabell

Did you find an answer? We are using Rack::Timeout. – Mimicry

Workaround was to scan the nginx logs for 503 (or some 50X, cant recall) status and see from there. Kinda meets the objective in some odd way. – Jessabell

Answer 1

I'd be inclined to create an account on New Relic, use their newrelic_rpm gem and collect performance data related to this issue in their web application. NR offer a free account level which is useful for this situation.

The benefit of using this somewhat lazy instrumentation method is twofold: no requirement for a custom instrumentation implementation in your application; and no need to check your downstream webserver logs for 503 server errors.

Starofbethlehem

Hi! Thanks for the answer! But i'm not looking to introduce a new service or an app at the moment. I looking for a specific hook into the timeout capture of unicorn – Jessabell

Then perhaps you should look at the 'configurator' and 'tuning' sections of the Unicorn manual. – Starofbethlehem

Answer 2

Refer to the 'configurator' section of the Unicorn manual website. There's a series of items around backlog and timeouts which should give you the control you need.

Starofbethlehem

I've gone thru those pages already. They do not talk about handling timeouts as far as I understand. They talk about controlling timeout period only. – Jessabell

Have you added an ensure block to the relevant parts of your application code which times out? – Starofbethlehem

If I knew all the slow parts I would just fix those parts and not care about timeout n all. I'm specifically only interested in handling timeout by getting informed. – Jessabell

Answer 3

You can add a middleware that logs this, just before the unicorn is getting killed (logs are not guaranteed to be processed if what you do in the middleware is slower than the 1 second interval, but you should still get a decent amount of visibility)

# config/initializers/log_unicorn_timeout.rb
class LogUnicornTimeout
  def initialize(app)
    @app = app
  end

  def call(env)
    t = Thread.new do
      sleep(UNICORN_TIMEOUT - 1) # just before the timeout kicks in - 1
      unless Thread.current[:done]
        path = env["PATH_INFO"]
        query = env["QUERY_STRING"]
        Rails.logger.warn "Path: #{path} Query: #{query}"
      end
    end

    @app.call(env)
  ensure
    t[:done] = true
    t.run
  end
end

Then add the middleware above in your application.rb

config.middleware.use 'LogUnicornTimeout'

Chatelaine