Sidekiq/Airbrake only post exception when retries extinguished

Question

I would like Airbrake to only be notified of errors when the retries are exhausted, but I can't seem to think of a way to implement it...

I can add a sidekiq_retries_exhausted hook to send the error to AirBrake but the only way I can think of catching the actual failures is to add a middleware that swallows the error, but then, the job will be marked as a success if there is no error... then there will never be any retries..

Hope that makes sense!

Oxbow

You could rescue and raise your own job error exception (managing the original error as you see fit) and then add that custom job error exception to the aibrake ignore list. Apologies if I've misunderstood your question. – Surtout

You understood correctly, however I'd rather the error didn't go to Airbrake at all... – Oxbow

To clarify, what I mean is, you wrap your perform code in a begin/rescue block, and then log any errors that arise that would have resulted in a retry, and instead raise a generic exception like WorkerPerformError and add that to your ignored exceptions list. That way the original exception is suppressed and the custom exception triggers a retry. I'm going to have to implement something similar to this on a recent project I inherited so I'll post a more thought-out solution when I have. – Surtout

Just found this thread which might help: librelist.com/browser//sidekiq/2012/12/8/… – Surtout

Answer 1

I managed to implement this with a Sidekiq middleware that is inserted at the start of the list:

class RaiseOnRetriesExtinguishedMiddleware
    include Sidekiq::Util

  def call(worker, msg, queue)
    yield
  rescue Exception => e
    bubble_exception(msg, e)
  end

  private

  def bubble_exception(msg, e)
    max_retries = msg['retries'] || Sidekiq::Middleware::Server::RetryJobs::DEFAULT_MAX_RETRY_ATTEMPTS
    retry_count = msg['retry_count'] || 0
    last_try = !msg['retry'] || retry_count == max_retries - 1

    raise e if last_try
  end

  def retry_middleware
    @retry_middleware ||= Sidekiq::Middleware::Server::RetryJobs.new
  end
end

If its the last try and its thrown an exception, it'll let it bubble up (to Airbrake) otherwise it won't. This doesn't affect failure recording as that happens later in the chain.

Oxbow

Answer 2

As shown here (not my code):

    Airbrake.configure do |config|
      config.api_key = '...'
      config.ignore_by_filter do |exception_data|
        exception_data[:parameters] && 
        exception_data[:parameters]['retry_count'].to_i > 0
      end
    end

Surtout

This is one way to do it, though I think my solution is cleaner, thanks though – Oxbow

This line is redundant: exception_data[:parameters]['retry_count'] && – Wulfenite

so it is. removed. – Surtout

Answer 3

I ran into the exact same thing, and wanted to keep it out of AirBrake. Here is what I did, which is easy to read and simple:

class TaskWorker
  include Sidekiq::Worker

  class RetryLaterNotAnError < RuntimeError
  end

  def perform task_id
    task = Task.find(task_id)
    task.do_cool_stuff

    if task.finished?
      @log.debug "Nothing to do for task #{task_id}"
      return false
    else
      raise RetryLaterNotAnError, task_id
    end
  end
end

And then, to get Airbrake to ignore it:

Airbrake.configure do |config|
  config.ignore << 'RetryLaterNotAnError'
end

Voila!

Fissirostral

Don't do this. This is terrible and not a proper solution. – Panter

I've found it met my needs quite well, but I'm always eager to learn. What is terrible about this? What would be a more proper solution? – Fissirostral

Sidekiq exposes a "retries_exhausted" block which allows you to handle the last exception of the last failure when there will be no more retries after that. We should use that instead. Any time you're using an exception for control flow, you should reevaluate. You're raising an exception as a workaround instead of properly supporting the logic. Sidekiq has logic for knowing when not to retry things, ergo, we should be able to hook into sidekiq to do things when sidekiq chooses not to retry things. – Panter

This workaround is not to implement retries_exhausted functionality. Rather, this allows my TaskWorker to signal sidekiq to retry later (with all the backoff/expiration candy it brings), without cluttering up my Airbrake logs with meaningless exceptions. The real driver here is that the only way to signal sidekiq to try again later is to raise an exception. That is not ideal (IMO), and there's a feature request to that end at github.com/mperham/sidekiq/issues/1704#issuecomment-390795666 – Fissirostral

Answer 4

Here is how we do it for Bugsnag, which you can customise for Airbrake.

# config/initializers/00_core_ext.rb
class StandardError
  def skip_bugsnag?
    !!@skip_bugsnag
  end

  def skip_bugsnag!
    @skip_bugsnag = true
    return self
  end
end

# config/initializers/bugsnag.rb

  config.ignore_classes << lambda { |e| e.respond_to?(:skip_bugsnag?) && e.skip_bugsnag? }

# In Sidekiq Jobs

raise ErrorToRetryButNotReport.new("some message").skip_bugsnag!

# Or if the error is raised by a third party

begin
  # some code that calls a third-party method
rescue ErrorToRetryButNotReport => e
  e.skip_bugsnag!
  raise
end

You can then manually choose to send the error from sidekiq_retries_exhausted.

Monochromatic