How to make ExceptionNotifier work with delayed_job in Rails 3?

Question

I'd like ExceptionNotifier to send out an email when an exception happens in a delayed job, just like for other exceptions. How can I achieve that?

Baal

Interesting question, in looking I came across #4104593 and groups.google.com/group/delayed_job/browse_thread/thread/… – Paraguay

Yep, I've seen both of those but I think these solutions only work with Rails 2. – Baal

Answer 1

I do this with Rails 3.2.6, delayed_job 3.0.3 and exception_notification 2.6.1 gem

# In config/environments/production.rb or config/initializers/delayed_job.rb

# Optional but recommended for less future surprises.
# Fail at startup if method does not exist instead of later in a background job 
[[ExceptionNotifier::Notifier, :background_exception_notification]].each do |object, method_name|
  raise NoMethodError, "undefined method `#{method_name}' for #{object.inspect}" unless object.respond_to?(method_name, true)
end

# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do 
  def handle_failed_job_with_notification(job, error)
    handle_failed_job_without_notification(job, error)
    # only actually send mail in production
    if Rails.env.production?
      # rescue if ExceptionNotifier fails for some reason
      begin
        ExceptionNotifier::Notifier.background_exception_notification(error)
      rescue Exception => e
        Rails.logger.error "ExceptionNotifier failed: #{e.class.name}: #{e.message}"
        e.backtrace.each do |f|
          Rails.logger.error "  #{f}"
        end
        Rails.logger.flush
      end
    end
  end 
  alias_method_chain :handle_failed_job, :notification 
end

It's probably a good idea to load this code in all environments to catch errors after bundle update etc before they reach production. I do this by having a config/initializers/delayed_job.rb file but you could duplicate the code for each config/environments/* environment.

Another tip is to tune the delayed job config a bit as default you may get a lot of duplicate exception mails when job fails.

# In config/initializers/delayed_job_config.rb
Delayed::Worker.max_attempts = 3

Update I had some problems with the delayed_job daemon silently exiting and it turned out to be when ExceptionNotifier fails to send mail and no one rescued the exception. Now the code rescues and log them.

Syncopate

I'd recommend using the official exception notification gem at github.com/smartinez87/exception_notification – Irenics

Ah ok, will update to that. I was using the rails3 gem as there seamd to be some rails 3 issues with the official version at the time. – Syncopate

+1 This approach works with Rails 3.2.6, Delayed Job 3.0.3, and Exception Notification 2.6.1. Initially I thought adding an error method using the built-in hook would be a cleaner approach, but that would have to be defined for each job class, and defining it generically for PerformableMethod didn't work due to how delegate is used. – Naamann

Nice. I actually updated to the same versions the other day but didn't remember to update the answer. – Syncopate

I've added a , :data => {:job => job} to the method call so that I have a bit more details… – Rania

To get notified ONLY when last attempt failed I add: ExceptionNotifier.notify_exception(error) if job.attempts == Delayed::Worker.max_attempts in place of: ExceptionNotifier::Notifier.background_exception_notification(error) – Thankless

Answer 2

Adding to @MattiasWadman answer, since exception_notification 4.0 there's a new way to handle manual notify. So instead of:

ExceptionNotifier::Notifier.background_exception_notification(error)

use

ExceptionNotifier.notify_exception(error)

Laid

will this also be processed in te background? – Subtotal

@DaniëlZwijnenburg The method is being called by delayed_job, so yes – Laid

Answer 3

Another way to handle exceptions (put as an initializer):

class DelayedErrorHandler < Delayed::Plugin

  callbacks do |lifecycle|

    lifecycle.around(:invoke_job) do |job, *args, &block|

      begin
        block.call(job, *args)
      rescue Exception => e

        # ...Process exception here...

        raise e
      end
    end
  end
end

Delayed::Worker.plugins << DelayedErrorHandler

P

I think this is cleanest solution. BTW other solutions wont work in the future, because alias_method_chain is deprecated in Rails 5 – Palenque

Answer 4

alias_method_chain no longer exists in Rails 5.

Here's the new (proper) way to do this using Ruby 2's prepend

# In config/initializers/delayed_job.rb
module CustomFailedJob
  def handle_failed_job(job, error)
    super
    ExceptionNotifier.notify_exception(error, data: {job: job})
  end
end

class Delayed::Worker
  prepend CustomFailedJob
end

Mcclanahan

This is the answer which worked for me, running Rails 4.2. – Prolix

Delayed::Worker.prepend CustomFailedJob can be used directly – Eastbound

Answer 5

For exception_notification 3.0.0 change:

ExceptionNotifier::Notifier.background_exception_notification(error)

to:

ExceptionNotifier::Notifier.background_exception_notification(error).deliver

Cheek

Answer 6

simpler and updated answer:

# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do
  def handle_failed_job_with_notification job, error
    handle_failed_job_without_notification job, error
    ExceptionNotifier.notify_exception error,
      data: {job: job, handler: job.handler} rescue nil
  end 
  alias_method_chain :handle_failed_job, :notification
end

And test on console with:

Delayed::Job.enqueue (JS=Struct.new(:a){ def perform; raise 'here'; end }).new(1)

Kinetic

@AlterLagos I think you misunderstand alias_method_chain. – Comeuppance