How do you run multiple DelayedJob workers on a single Heroku dyno?

Question

I am having trouble getting my dynos to run multiple delayed job worker processes.

My Procfile looks like this:

worker: bundle exec script/delayed_job -n 3 start

and my delayed_job script is the default provided by the gem:

#!/usr/bin/env ruby

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize

When I try to run this either locally or on a Heroku dyno it exits silently and I can't tell what is going on.

foreman start
16:09:09 worker.1 | started with pid 75417
16:09:15 worker.1 | exited with code 0
16:09:15 system   | sending SIGTERM to all processes
SIGTERM received

Any help with either how to debug the issue or suggestions about other ways to go about running multiple workers on a single dyno it would be greatly appreciated.

Permanency

Answer 1

You can use foreman to start multiple processes on the same dyno.

First, add foreman to your Gemfile.

Then add a worker line to your Procfile:

worker: bundle exec foreman start -f Procfile.workers

Create a new file called Procfile.workers which contains:

dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work

That will start 3 delayed_job workers on your worker dyno.

Phonate

Answer 2

Try changing your Procfile to:

worker: bundle exec script/delayed_job -n 3 run

Using start will create two daemons in the background and then immediately exit. Heroku thinks that your process crashed.

Using run keeps the workers in the foreground.

UPDATE: I use Delayed Job Worker Pool for this now.

Popelka

Yes, been looking all over for this. This should be the accepted answer. – Bathyal

Sorry to say this doesn't appear to work anymore. Run will now ignore the -n arguments and always run a single process. – Crackling

Answer 3

Short answer is that you can't do this with delayed_job. A dyno is a process and a single delayed_job worker works on a single process.

There are other solutions to this though. If you can switch over to using Sidekiq then you can run quite a few workers on a single process since Sidekiq workers use threading. The trade-off here is that your workers will need to be thread safe.

Check it out: http://sidekiq.org/

Expostulation

Answer 4

I tried to use the accepted solution of @infused, but ran into an issue with the worker processes crashing on startup with Bundler::GemNotFound: Unable to find a spec satisfying web-console in the set. Perhaps the lockfile is corrupted?

The reason for that is that foreman start is looking for a .env file by default and read it: Forman Manual. Because the .env file was setup for a development environment, foreman was looking for the web-console gem and missing it. Conslusion: If you see your worker not starting, check if you have a .env file that is unsuited for production and rename it or specify another file on startup.

Dripping

Answer 5

I used the ruint buildpack to run a Procfile.worker in a single (worker) dyno: https://github.com/heroku/heroku-buildpack-runit

Piscina