2014-03-14 twitter

cold hands, warm heart

Zero Downtime Deployment With Unicorn And Capistrano

Everytime we're fixing a bug or adding a new feature, we want it to go live very fast to have happy users at gigmit. Thats why we really love to deploy several times a day. While running a pretty standard Rails stack with Unicorn and handling deployments with Capistrano, we always had a few seconds when the Unicorn workers were restarted. We did this only at night, because it wasn't always sure[1], if they will really come back without errors.

So I searched the internet for how others do this with Unicorn and found many different deploy scripts and blog posts . I tried to get my head around all this and basically our scripts we're using now are a compilation of all the stuff I found there. Plus because I don't really like bash scripting, I tried to do as much as possible in Ruby.

Basically there are three files involved in the whole process. The Unicorn config file, which is checked in in our app's git repo; an init.d script for handling unicorn as a service on linux and our app's config/deploy.rb because Capistrano wants you to overwrite their empty restart task to let it know, what to do at the end of a successful deployment.

The Unicorn config is at config/unicorn.rb and has the following content:

root = "/home/deployer/current"

pid           "#{root}/tmp/pids/unicorn.pid"
stderr_path   "#{root}/log/unicorn_error.log"
stdout_path   "#{root}/log/unicorn.log"

listen "/tmp/unicorn.gigmit.com.socket", :backlog => 2048

preload_app true
working_directory root
worker_processes 3
timeout 30

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!

  # When sent a USR2, Unicorn will suffix its pidfile with .oldbin and
  # immediately start loading up a new version of itself (loaded with a new
  # version of our app). When this new Unicorn is completely loaded
  # it will begin spawning workers. The first worker spawned will check to
  # see if an .oldbin pidfile exists. If so, this means we've just booted up
  # a new Unicorn and need to tell the old one that it can now die. To do so
  # we send it a QUIT.
  # Using this method we get 0 downtime deploys.

  old_pid = "#{root}/tmp/pids/unicorn.pid.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us

after_fork do |server, worker|
  # Unicorn master loads the app then forks off workers - because of the way
  # Unix forking works, we need to make sure we aren't using any of the parent's
  # sockets, e.g. db connection

  defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
  # Redis and Memcached would go here but their connections are established
  # on demand, so the master never opens a socket

  # Unicorn master is started as root, which is fine, but let's
  # drop the workers to deployer:deployer

    uid, gid = Process.euid, Process.egid
    user, group = 'deployer', 'deployer'
    target_uid = Etc.getpwnam(user).uid
    target_gid = Etc.getgrnam(group).gid
    worker.tmp.chown(target_uid, target_gid)
    if uid != target_uid || gid != target_gid
      Process.initgroups(user, target_gid)
  rescue => e
    if RAILS_ENV == 'development'
      STDERR.puts "couldn't change user, oh well"
      raise e

Beside some other stuff, the important part happens in the before_fork method. If a worker came up successfully and is accepting new requests (which means your app is ok after a change) it checks if there is an old PID. If yes, the new worker will tell the old process to (finish all pending requests and) quit. This means, only if we get a new version of the app up and running, we kill the old one. So if something fails in the startup process, the old version of the app will still be available.

To handle Unicorn with init.d, we have this script in /etc/init.d/unicorn:

# Provides:          unicorn
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Manage unicorn server
# Description:       Start, stop, restart unicorn server for a specific application.
set -e

# Feel free to change any of the following variables for your app:
export RUBY_GC_HEAP_MIN_SLOTS=800000
export RUBY_GC_MALLOC_LIMIT=50000000

CMD="cd $APP_ROOT && BUNDLE_GEMFILE=$APP_ROOT/Gemfile bundle exec unicorn -D -c $APP_ROOT/config/unicorn.rb -E production"
set -u


sig () {
  test -s "$PID" && kill -s $1 `cat $PID`

run () {
  if [ "$(id -un)" = "$AS_USER" ]; then
    eval $1
    su -c "$1" - $AS_USER

case "$1" in
  sig 0 && echo >&2 "Already running" && exit 0
  run "$CMD"
  sig QUIT && exit 0
  echo >&2 "Not running"
  sig TERM && exit 0
  echo >&2 "Not running"
  sig HUP && echo reloaded OK && exit 0
  echo >&2 "Couldn't reload, starting '$CMD' instead"
  run "$CMD"
  if sig USR2 && sleep 3
    while test -s $OLD_PIN && test $n -ge 0
      printf '.' && sleep 1 && n=$(( $n - 1 ))

    if test $n -lt 0 && test -s $OLD_PIN
      echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
      exit 1
    exit 0
  echo >&2 "Couldn't upgrade, starting '$CMD' instead"
  run "$CMD"
  sig USR1
  echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
  exit 1

On a deployment we only call service unicorn upgrade which is sending a USR2 signal at the current Unicorn process. After this it will wait 60 seconds for the old PID to disappear, which from what we learned above only happens if a new process came up successfully otherwise it will fail and interrupt the deployment.

To let Capistrano handle all of this stuff automatically I added the following to our config/deploy.rb:

namespace :deploy do
  desc "restart (upgrade) unicorn server"
  task :restart, roles: :app, except: {no_release: true} do
    run "service unicorn upgrade"

Which means we now have zero downtime deployment with a simple cap production deploy. Woop Woop!

PS: We had some problems starting new Unicorn workers with a new set of Gem versions. We fixed that by explicitly specifying the Gemfile in the init.d script.

[1] We have a staging environment, but its not 100% an exact copy of our production system, because of some payment provider foo. But thats probably worth another blogpost.

2014-03-08 twitter

wenn programme an karma glauben

2014-03-07 twitter

amor fati

2014-03-05 twitter

updated my #puppet module for #rubybuild, added easier update handling and a static #ruby path option! check it out: http://t.co/sb9Pxkkpre