Starting multiple upstart instances automatically

Question

We use PHP gearman workers to run various tasks in parallel. Everything works just fine, and I have silly little shell script to spin them up when I want them. Being a programmer (and therefore lazy), I wanted to see if I could spin these up via an upstart script.

I figured out how to use the instance stanza, so I could start them with an instance number:

description "Async insert workers"
author      "Mike Grunder"

env SCRIPT_PATH="/path/to/my/script"

instance $N

script
    php $SCRIPT_PATH/worker.php
end script

And this works great, to start them like so:

sudo start async-worker N=1
sudo start async-worker N=2

The way I want to use these workers is to spin up some number of them (maybe one per core, etc), and I would like to do this on startup. To be clear, I don't need the upstart script to detect the number of cores. I'm happy to just say "do 8 instances", but that's why I want multiple running. Is there a way for me to use the "start on" clause in an upstart script to do this automatically?

For example, start instance 1, 2, 3, 4? Then have them exit on shutdown properly?

I suppose I could hook this into an init.d script, but I was wondering if upstart can handle something like this, or if anyone has figured out this issue.

Cheers guys!

Rinehart

Answer 1

What you need is a bootstrap task that runs on startup and iterates over all your worker jobs, starting each one.

#/etc/init/async-workers-all.conf

start on runlevel [2345]

task

env NUM_WORKERS=8

script
  for i in `seq 1 $NUM_WORKERS`
  do
    start async-worker N=$i
  done
end script

The key is to make this a task, which tells upstart to let the task run to completion before emitting any events for it. See http://upstart.ubuntu.com/cookbook/#task and http://upstart.ubuntu.com/cookbook/#instance

Microscopium

Thanks so much! I tried to work it out in a similar manner, but I was missing the task bit, so the first iteration of the loop would just block. Awesome. :) – Rinehart

You should also checkout #12084525 that allows to not only start multiple instances, but also stop them. – Conventioner

I've read the documentation but I still don't get it, why is that loop blocking in its first iteration unless it's a task? /cc @Rinehart – Postern