God starts too many processes

Question

I have a god script that is supposed to keep an eye on two stalker processes. The problem is that after 24h it starts way too many processes.

This is the god script.

rails_root = File.expand_path("../..", __FILE__)

2.times do |n|
  God.watch do |w|
    w.group = "app-scripts"
    w.name  = "run-#{n}"
    w.interval = 30.seconds
    w.dir      = File.dirname(__FILE__)

    w.env = {
      "BUNDLE_GEMFILE" => "#{rails_root}/Gemfile",
      "RAILS_ENV" => "production",
      "BEANSTALK_URL" => "beanstalk://127.0.0.1:54132"
    }

    w.start = "bbundle exec stalk #{File.join(rails_root, "config/jobs.rb")}"

    w.start_grace = 5.seconds
    w.stop_grace  = 5.seconds

    w.start_if do |start|
      start.condition(:process_running) { |c| c.running = false }
    end

    w.restart_if do |restart|
      restart.condition(:memory_usage) do |c|
        c.above = 200.megabytes
        c.times = [3, 5]
      end

      restart.condition(:cpu_usage) do |c|
        c.above = 95.percent
        c.times = 5
      end
    end

    w.lifecycle do |on|
      on.condition(:flapping) do |c|
        c.to_state = [:start, :restart]
        c.times = 5
        c.within = 5.minute
        c.transition = :unmonitored
        c.retry_in = 10.minutes
        c.retry_times = 5
        c.retry_within = 2.hours
      end
    end
  end
end

ps aux | grep stalk returns the following.

root      3178  0.2  2.7 417580 117284 ?       Sl   Oct28   2:22 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root      3179  0.2  3.3 506068 138740 ?       Sl   Oct28   2:26 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root      4588  0.2  2.9 497932 121664 ?       Sl   Oct25  16:10 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root      4794  0.2  3.0 497792 128084 ?       Sl   Oct25  15:57 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     10391  0.2  2.8 496784 121388 ?       Sl   Oct25  15:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     10392  0.2  2.8 497624 121528 ?       Sl   Oct25  15:31 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     18874 75.0  2.0 214116 83948 ?        Rl   15:49   0:09 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     18875 75.0  2.0 214944 84868 ?        Rl   15:49   0:09 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     20649  0.2  2.6 410636 110012 ?       Sl   Oct28   2:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     20650  0.2  3.0 439284 128996 ?       Sl   Oct28   2:47 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     23272  0.2  2.7 414452 115772 ?       Sl   Oct28   2:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     23273  0.2  2.7 417728 117152 ?       Sl   Oct28   2:44 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     25919  0.2  3.1 436276 131876 ?       Sl   Oct28   2:28 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     25920  0.2  3.3 503236 138676 ?       Sl   Oct28   2:29 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     28782  0.2  2.8 431836 121108 ?       Sl   Oct25  16:58 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     30687  0.2  2.7 415908 117008 ?       Sl   Oct28   2:39 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb
root     30688  0.2  2.6 476184 111844 ?       Sl   Oct28   2:37 ruby /opt/www/myapp/shared/bundle/ruby/1.9.1/bin/stalk /opt/www/myapp/current/config/jobs.rb

This is the /usr/bin/bbundle script.

#!/usr/bin/env bash

if [[ -s "/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster" ]]
then
  source "/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster"
  bundle  "$@"
else
  echo "ERROR: Missing RVM environment file: '/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster'" >&2
  exit 1
fi

• Running sudo god stop app-scripts won't kill any processes.

• I've tried adding w.uid = "webmaster" to the god script, but the problem remains.

• I'm running god version 0.12.1, ruby version 1.9.3p286 and stalker version 0.9.0.

What am I doing wrong?

Seraphic

Answer 1

I solved the problem thanks to mpapis @ the rvm irc. I had to changed line 6 in the bbundle script from bundle "$@" to exec bundle "$@".

#!/usr/bin/env bash

if [[ -s "/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster" ]]
then
  source "/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster"
  exec bundle  "$@"
else
  echo "ERROR: Missing RVM environment file: '/home/webmaster/.rvm/environments/ruby-1.9.2-p320@webmaster'" >&2
  exit 1
fi

Seraphic

Answer 2

It seems like god is trying to follow bbundle and not stalk. You need to let god know where to find the PID of the actual process you want to follow with w.pid_file. You might also need to tell it how to kill the process, if a standard kill won't do the trick. For that you can use w.stop_signal for a different signal (as simonmenke suggested) or w.stop for a whole other command.

The log file should shed more light on what's going on. Call god -D to print it to stdout or god -l /var/log/god.log.

Foraminifer

This is the output of god. gist.github.com/dcac99caec1b712c1817 Yeah, it looks like god is killing /usr/bin/bbundle not ruby stalk. I'll guess I've to specify where the pid file is placed. – Seraphic

It looks like it isn't possible to specify a pid file when using stalker. What should I do now ? I'm not sure how to proceed. – Seraphic

Well, we can get creative. You can have the first job in line write the PID to a file and refer god to that file. Or just write your own version of bin/stalk that writes the PID file before running through the job queue. – Foraminifer

One solution would be to let stalk run File.open("pid-file.pid", "w") { |file| file.write($$.to_s) }. I'll do some research. – Seraphic

Your god script suggests you want two processes of stalker running. So you may want to accept a PID file path in the command line and give each of those two stalkers a different one. – Foraminifer

I tried adding this to the top of my config/jobs.rb file File.open(ENV["PID"], "w") { |file| file.write($$.to_s) }. Then I added w.env = { "PID" => pid_file }and w.pid_file = pid_file, but the problem remains. It still doesn't kill any of the processes when running god stop app-scripts. – Seraphic

What is pid_file? Where is it defined? Please attach the new god log. – Foraminifer

I solved it thanks to mpapis @ rvm irc. This is the new bbundle script gist.github.com/8eca8688f2caef15c595. I changed line 6 from bundle "$@" to exec bundle "$@". – Seraphic

Answer 3

Stacker stops when it receives a INT signal (not a TERM signal). Try adding a stop signal:

# ...
w.stop_signal = 'INT'
# ...

Reed

That didn't work. I tried to run sudo god restart myapp, but it only started one new process, but didn't kill the old one. – Seraphic

Do the stalker processes detach (demonize) them selfs? If so you also need to tell god where to find the pid. – Reed

No, stalker doesn't demonize it self. That should't be necessary. – Seraphic

Answer 4

This should help with your question: Monitor a Rake task with God.

In short, you can store a reference to your PID file in your god config:

 God.watch do |w|
   w.dir = "#{rails_root}"
   w.name = "my_task"
   w.interval = 10.seconds
   w.pid_file = "#{rails_root}/tmp/pids/#{w.name}.pid"
   w.env = {"RAILS_ENV"=>rails_env, 'PIDFILE' => w.pid_file}
   w.start = "bundle exec rake my_task &"
   ...
 end

and in your running process you write to this file your PID (in this example, rake):

 task :my_task => :environment do
   File.open(ENV['PIDFILE'], 'w') { |f| f << Process.pid } if ENV['PIDFILE']
   Model.perform_task!
 end

We pass the path of the PID file god is watching to the process that is actually being watched and that writes its PID to that file (which god then monitors). Hope this helps.

Meerkat