Starting or restarting Unicorn with Capistrano 3.x

Question

I'm trying to start or restart Unicorn when I do cap production deploy with Capistrano 3.0.1. I have some examples that I got working with Capistrano 2.x using something like:

namespace :unicorn do
  desc "Start unicorn for this application"
  task :start do
    run "cd #{current_path} && bundle exec unicorn -c /etc/unicorn/myapp.conf.rb -D"
  end
end

But when I try and use run in the deploy.rb for Capistrano 3.x I get an undefined method error.

Here are a couple of the things I tried:

# within the :deploy I created a task that I called after :finished
namespace :deploy do
...

  task :unicorn do
    run "cd #{current_path} && bundle exec unicorn -c /etc/unicorn/myapp.conf.rb -D"
  end

  after :finished, 'deploy:unicorn'

end

I have also tried putting the run within the :restart task

namespace :deploy do
  desc 'Restart application'
  task :restart do

  on roles(:app), in: :sequence, wait: 5 do
    # Your restart mechanism here, for example:
    # execute :touch, release_path.join('tmp/restart.txt')
    execute :run, "cd #{current_path} && bundle exec unicorn -c /etc/unicorn/deployrails.conf.rb -D"
  end
end    

If I use just run "cd ... " then I'll get awrong number of arguments (1 for 0)` in the local shell.

I can start the unicorn process with unicorn -c /etc/unicorn/deployrails.conf.rb -D from my ssh'd VM shell.

I can kill the master Unicorn process from the VM shell using kill USR2, but even though the process is killed I get an error. I can then start the process again using unicorn -c ...

$ kill USR2 58798
bash: kill: USR2: arguments must be process or job IDs

I'm very new to Ruby, Rails and Deployment in general. I have a VirtualBox setup with Ubuntu, Nginx, RVM and Unicorn, I'm pretty excited so far, but this one is really messing with me, any advice or insight is appreciated.

Chiastic

Answer 1

Can't say anything specific about capistrano 3(i use 2), but i think this may help: How to run shell commands on server in Capistrano v3?. Also i can share some unicorn-related experience, hope this helps.

I assume you want 24/7 graceful restart approach.

Let's consult unicorn documentation for this matter. For graceful restart(without downtime) you can use two strategies:

1. kill -HUP unicorn_master_pid It requires your app to have 'preload_app' directive disabled, increasing starting time of every one of unicorn workers. If you can live with that - go on, it's your call.

2. kill -USR2 unicorn_master_pid kill -QUIT unicorn_master_pid

More sophisticated approach, when you're already dealing with performance concerns. Basically it will reexecute unicorn master process, then you should kill it's predecessor. Theoretically you can deal with usr2-sleep-quit approach. Another(and the right one, i may say) way is to use unicorn before_fork hook, it will be executed, when new master process will be spawned and will try to for new children for itself. You can put something like this in config/unicorn.rb:

# Where to drop a pidfile
pid project_home + '/tmp/pids/unicorn.pid'

before_fork do |server, worker|
  server.logger.info("worker=#{worker.nr} spawning in #{Dir.pwd}")

  # graceful shutdown.
  old_pid_file = project_home + '/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid_file) && server.pid != old_pid_file
    begin
      old_pid = File.read(old_pid_file).to_i
      server.logger.info("sending QUIT to #{old_pid}")
      # we're killing old unicorn master right there
      Process.kill("QUIT", old_pid)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

It's more or less safe to kill old unicorn when the new one is ready to fork workers. You won't get any downtime that way and old unicorn will wait for it's workers to finish.

And one more thing - you may want to put it under runit or init supervision. That way your capistrano tasks will be as simple as sv reload unicorn, restart unicorn or /etc/init.d/unicorn restart. This is good thing.

Interact

Thanks dredozubov! I'm getting closer. The link and checking out sshkit were really helpful. I'm able to start with a deploy when there isn't a pid file and I can have a kill -USR2 $(< #{unicorn_pid}) && kill -QUIT $(< #{unicorn_pid}) work once, but if I run the deploy again the USR2 doesn't re-execute and I have the same PID integer. Then that duplicate PID seems kill the app or at least give me a blank page. Any ideas? I'm going to keep trying. – Chiastic

I think it's some kind of capistrano symlink issue. Capistrano links /current to some exact version. You should check it out closely. It may be kind of elusive bug. – Interact

Yes it's elusive! :) One thing I'm noticing is that once I have the bug happen where the PID doesn't clear I'm unable to do it in the shell with the usual kill -USR2 <pid>, the same master and worker PIDs just keeps showing up when I do ps ax | grep unicorn. (I can use -QUIT and remove the master). I'm going to keep playing around, thanks this is a great learning experience and I'm getting more comfortable in the shell! – Chiastic

I think my problem is that I keep running cap production deploy without really making any updates or pushes to the repo, in the real world someone probably never does that right? Because eventually, when I work through the release number the unicorn.pid doesn't change. I made a gist that shows my deploy.rb and my unicorn.log. Thanks dredozubov, my original question has been resolved with how to gracefully restart (seems like I can get some zero downtime going). – Chiastic

You can push on cap production deploy if you really want to. Congrats, it could be trickier than that. That seems kinda obvious to me, so i didn't even thought of it. I guess you're working solo, cause in team environment you always push your changes to some 'hub' repo. – Interact

Answer 2

I'm using following code:

namespace :unicorn do
  desc 'Stop Unicorn'
  task :stop do
    on roles(:app) do
      if test("[ -f #{fetch(:unicorn_pid)} ]")
        execute :kill, capture(:cat, fetch(:unicorn_pid))
      end
    end
  end

  desc 'Start Unicorn'
  task :start do
    on roles(:app) do
      within current_path do
        with rails_env: fetch(:rails_env) do
          execute :bundle, "exec unicorn -c #{fetch(:unicorn_config)} -D"
        end
      end
    end
  end

  desc 'Reload Unicorn without killing master process'
  task :reload do
    on roles(:app) do
      if test("[ -f #{fetch(:unicorn_pid)} ]")
        execute :kill, '-s USR2', capture(:cat, fetch(:unicorn_pid))
      else
        error 'Unicorn process not running'
      end
    end
  end

  desc 'Restart Unicorn'
  task :restart
  before :restart, :stop
  before :restart, :start
end

Blackington

Answer 3

I'm just going to throw this in the ring: capistrano 3 unicorn gem

However, my issue with the gem (and any approach NOT using an init.d script), is that you may now have two methods of managing your unicorn process. One with this cap task and one with init.d scripts. Things like Monit / God will get confused and you may spend hours debugging why you have two unicorn processes trying to start, and then you may start to hate life.

Currently I'm using the following with capistrano 3 and unicorn:

  namespace :unicorn do
  desc 'Restart application'
    task :restart do
      on roles(:app) do
        puts "restarting unicorn..."
        execute "sudo /etc/init.d/unicorn_#{fetch(:application)} restart"
        sleep 5
        puts "whats running now, eh unicorn?"
        execute "ps aux | grep unicorn"
      end
    end
end

The above is combined with the preload_app: true and the before_fork and after_fork statements mentioned by @dredozubov

Note I've named my init.d/unicorn script unicorn_application_name.

The new worker that is started should kill off the old one. You can see with ps aux | grep unicorn that the old master hangs around for a few seconds before it disappears.

Holyhead

thanks this work for me :-) after 2 days of configuration. – Meshuga

Answer 4

To view all caps:

cap -T

and it shows:

***
cap unicorn:add_worker             # Add a worker (TTIN)
cap unicorn:duplicate              # Duplicate Unicorn; alias of unicorn:re...
cap unicorn:legacy_restart         # Legacy Restart (USR2 + QUIT); use this...
cap unicorn:reload                 # Reload Unicorn (HUP); use this when pr...
cap unicorn:remove_worker          # Remove a worker (TTOU)
cap unicorn:restart                # Restart Unicorn (USR2); use this when ...
cap unicorn:start                  # Start Unicorn
cap unicorn:stop                   # Stop Unicorn (QUIT)
***

So, to start unicorn in production:

cap production unicorn:start

and restart:

cap production unicorn:restart 

PS do not forget to correct use gem capistrano3-unicorn

https://github.com/tablexi/capistrano3-unicorn

Halfpint

Answer 5

You can try to use native capistrano way as written here:

If preload_app:true and you need capistrano to cleanup your oldbin pid use:

after 'deploy:publishing', 'deploy:restart'
namespace :deploy do
  task :restart do
    invoke 'unicorn:legacy_restart'
  end
end

Tideway