How to limit the number of instances UpStart can simultaneously start or respawn

Question

I have a UpStart service job that has many (~100) instance that need to be started. Each of them is a resource-heavy process that does a lot of disk reading/writing during startup. When all of them start or respawn at the same time, they cause trouble due to excessive disk read/write requests.

I need a way to limit the number of instances that UpStart tries to start or respawn simultaneously. For example, is there a way to let UpStart hold off launching another instance until, say 30 seconds, after the startup or respawning of another instance has begun?

Kress

If your job is a shell script, you could insert either a random pause, so that when 10 start at the same time they each pause a different number of seconds and space themselves out -- or you could have all jobs lock a certain file when doing the intensive startup. But that is a shell scripting solution that has nothing to do with upstart. – Thurston

can you show a upstart script? are they al somewhat the same? – Lively

I think the lock method mentioned by @Thurston is cool. But that'll require changes to the app code itself (btw, it is not a shell script. It is a full compiled application). It is surprising that UpStart doesn't have some builtin functionality to automatically hold and queue service instances (?). – Kress

Answer 1

You can start them in sequence by using

start on started otherUpstartService

Lively

When is a service considered "started" by UpStart? As soon as the process is launched? If it is too soon, it might not help with spreading the load out in time. If it is too late, there may be too much delay. – Kress

Answer 2

You can use pre-start or post-stop to pause after each job. E.g post-stop exec sleep 5

Deckle

If they all sleep for the same amount of time at start up, then the issue is still there. Is there a way to make the sleep time random? – Kress

You could do something like post-stop exec sleep $(( ( RANDOM % 10 ) + 1 )) to get sleeps between 1 and 10 seconds. Just change the 10 to any other integer to get a wider range – Deckle