Using Sidekiq for Active Job and getting ActiveJob::DeserializationError

Question

I'm trying to use Sidekiq to run the below job.

The job performs fine when not queued (perform_now) but fails when called as (perform_later), which uses Sidekiq.

AddEmployeesToRoomJob.perform_now room  ## works fine
AddEmployeesToRoomJob.perform_later room  ## breaks in Sidekiq

Error:

AddEmployeesToRoomJob JID-da24b13f405b1ece1212bbd5 INFO: fail: 0.003     sec
2016-08-20T14:57:16.645Z 19456 TID-owmym5fbk WARN:     {"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped"    :"AddEmployeesToRoomJob","queue":"default","args":    [{"job_class":"AddEmployeesToRoomJob","job_id":"0ba5bd30-e281-49a7-a93f-    6e50445183ac","queue_name":"default","priority":null,"arguments":    [{"_aj_globalid":"gid://dragonfly/Room/1"}],"locale":"en"}],"retry":true,    "jid":"da24b13f405b1ece1212bbd5","created_at":1471704675.739077,"enqueued    _at":1471705036.6406531,"error_message":"Error while trying to     deserialize arguments: Couldn't find Room with     'id'=1","error_class":"ActiveJob::DeserializationError","failed_at":14717    04675.946183,"retry_count":4,"retried_at":1471705036.644416}
2016-08-20T14:57:16.645Z 19456 TID-owmym5fbk WARN:     ActiveJob::DeserializationError: Error while trying to deserialize     arguments: Couldn't find Room with 'id'=1
2016-08-20T14:57:16.645Z 19456 TID-owmym5fbk WARN:     /Users/tamlyn/.rvm/gems/ruby-2.2.3/gems/activerecord-    5.0.0.1/lib/active_record/relation/finder_methods.rb:357:in     `raise_record_not_found_exception!'

My Job class AddEmployeesToRoomJob < ApplicationJob queue_as :default

  def perform(room)
    employees = Employee.all
    if employees.length > 0
      employees.each do |employee|
        UserRoom.create(user: employee, room: room)
      end
    end
  end
end

My Thoughts I don't understand why it can't find the room which I'm passing into the perform method. It's as though it somehow loses that variable in the queueifying / JSONifying of the job?

The Sidekiq docs say

"Unfortunately this means that if the [Room] record is deleted after the job is enqueued but before the perform method is called, exception handling is different."

They suggest a workaround but I don't see how that would help me:

rescue_from ActiveJob::DeserializationError do |exception|
    # handle a deleted user record
end

Thanks in advance for any help!

Dilate

github.com/mperham/sidekiq/wiki/… – Wagshul

Answer 1

I don't think it is a good idea to pass the Room object into a Sidekiq worker. I've always passed the primary key for a database object and then re-queried. Try this.

AddEmployeesToRoomJob.perform_later room.id

def perform(room_id)
  room = Room.find(room_id)
  employees = Employee.all
    if employees.length > 0
      employees.each do |employee|
        UserRoom.create(user: employee, room: room)
      end
    end
  end
end

Bipartisan

@TamlynR if this answer solves your problem, would you mind making it as the answer. If not, let me know what else is needed. – Bipartisan

Why is this needed though? – Prato

It's not needed but it prevents a lot of issues like the object being deleted or modified after being queued and before being processed. In most scenarios, you want to act on the most recent version of the object. – Bipartisan

Answer 2

Stumbled upon this myself and found the method discard_on very useful. Mostly it is very unnecessary to do jobs on deleted or never created records.

Example:

class ApplicationJob < ActiveJob::Base
  discard_on ActiveJob::DeserializationError do |job, error|
    Rails.logger.error("Skipping job because of ActiveJob::DeserializationError (#{error.message})")
  end
end

Perineum

Exactly what I was looking for. Thank you! – Eringo

Answer 3

You can pass a model to a job if you can ensure that this model has been commited. It's a classic error to enqueue jobs before commit the current transaction.

You can find more information here: https://github.com/mperham/sidekiq/wiki/FAQ#why-am-i-seeing-a-lot-of-cant-find-modelname-with-id12345-errors-with-sidekiq

In our project we coded a model concern to be able to add dynamic after commit code blocks.

module AfterCommitOnce
  extend ActiveSupport::Concern

  included do
    after_commit :execute_after_commit_handlers
  end

  def after_commit_once(&block)
    @after_commits = @after_commits || []
    @after_commits << block
  end

  private

  def execute_after_commit_handlers
    @after_commits = @after_commits || []
    @after_commits.each do |ac|
      ac.call
    end
    @after_commits = []
  end
end

Adding this concern to you Employee model to you can specify which job you want to execute after commit all changes to you database with:

...
employee.after_commit_once do
   AddEmployeesToRoomJob.perform_later
end
employee.save!

Gin

Answer 4

It's not advisable to send Ruby object as parameter to sidekick worker. You can send Id as parameter and initialise the object inside perform method. If want to send object then you can dump Ruby object to any other format like Json/ Binary / yml as follows.

object.to_json 

Or

Marshal.dump(object)

And before using the objects inside perform method you can de-serialize the object to Ruby object as follows.

JSON.parse(serialized_object)

Or

Marshal.load(serialized_object)

These are solutions for your problem, but not ideal solution.

Mythos

Thanks, that's really interesting – Dilate

Answer 5

On my case, this was a Ruby subclass issue, I have an OutgoingTextMessage class that inherits from the TextMessage class, when I enqueue with:

SendMessageJob.perform_later(self)  

I got the ActiveJob::DeserializationError error because Sidekiq can't find "OutgoingTextMessage.find" because the model where the id exists is on the TextMessage model. Using:

SendMessageJob.perform_later(id)

and then reading from the correct model, fixed the problem.

I don't understand why Sidekiq can't find the record though, if I try on the console:

ogtm = OutgoingTextMessage.find 83247

all works fine. This is not a race condition issue because even putting the MailerJob in a method called after the callback "after_create_commit" and adding:

sleep 9 

still Sidekiq can't find the record, but passing the Model.id fixed the problem.

Anyways, lesson learnt: don't save objects on Sidekiq, only ids.

Brunei