Rails Async Active Job doesn't execute code, while inline does

Question

Does the :async queue adapter actually do anything?

:inline, which is what is default in Rails 4, processes jobs built with ActiveJob, uh... inline, in the current execution thread. Async, shouldn't. It should use the ConnectionPool to not run it in the current thread, and that's ideally what would be happening. It'd run perform outside of the current execution thread.

But nothing executes it.

I've pored through the docs, and the only thing I can fathom is is that :async, unlike :inline, doesn't execute tasks, and expects you to build a system around execution locally. I have to manually perform perform on all jobs in order to get them to execute locally. When I set the adapter to :inline, it works just fine without having to execute.

Is there some configuration issue I'm missing that's preventing async from working correctly (like ActionCable?).

Does it not work if executed from a rake task (or the console?).

It works fine with :sidekiq/:resque, but I don't want to be running these locally all the time.

Rails by default comes with an "immediate runner" queuing implementation. That means that each job that has been enqueued will run immediately.

This is kind of what's cueing me in there being something wrong. I have jobs that are sitting in a queue somewhere that just don't run. What could be stopping this?

Ptosis

Answer 1

This is what I discovered. With the advent of concurrent-ruby, rake tasks aren't set up to handle this.

If you read in the documentation, it says with :async, it's cleared out of memory when the process ends.

Rails itself only provides an in-process queuing system, which only keeps the jobs in RAM. If the process crashes or the machine is reset, then all outstanding jobs are lost with the default async back-end.

Rake processes end when they're over. So, if you're doing any sort of data changes, rake tasks won't be open long enough to run a job, which is why they run :inline just fine, but not :async.

So, I haven't figured out a solution to keep rake tasks open long enough to run something :async (and keep the app :async the entire time). I have to switch it to :inline to run tasks, and then back to :async when I'm done for the rest of my jobs. It's why it works fine with :sidekiq or :resque, because those applications keep the job information in memory and do not release when the rake task is over.

In order for rake tasks to work with :async locally, there's not much you can do other than run tasks as :inline if you're local until rake (as a task runner) understands how to stay open while asynchronous tasks have been launched (or not). As a development only feature, this isn't really high priority, but, if you're bashing your head on the table not understanding why :async by default tasks that run jobs won't actually run, that's why.

Ptosis

@januszm glad I could get you a little assistance. – Ptosis

Answer 2

Here's what you can put at the end of a rake task to wait for the AsyncAdapter to finish processing before exiting the rake task:

if Rails.configuration.active_job.queue_adapter == :async
  executor =
    ActiveJob::Base._queue_adapter.instance_eval do
      @scheduler.instance_eval { @async_executor }
    end
  sleep(0.1) while executor.scheduled_task_count > executor.completed_task_count
end

Glebe