Rails + Puma + Bunny - Timeout error

Question

When set rabbitmq connection in initializer

#config/initializers/rabbitmq.rb
$rabbitmq_connection = Bunni.new "amqp://#{user}:#{pass}@#{host}:#{port}#{vhost}"
$rabbitmq_connection.start
$rabbitmq_channel = $rabbitmq_connection.create_channel

than thrown Timeout error from the place where I try to create exchange and publish

class Publisher
...
  exchange = $rabbitmq_channel.topic 'some', {auto_delete: false, passive: true}
end

Error trace

E, [2015-10-05T11:59:16.448537 #14889] ERROR -- : Error: Timeout::Error: Timeout::Error from
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/concurrent/continuation_queue.rb:33:in `block in poll'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/concurrent/continuation_queue.rb:30:in `synchronize'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/concurrent/continuation_queue.rb:30:in `poll'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/channel.rb:1774:in `wait_on_continuations'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/channel.rb:1176:in `block in exchange_declare'
/usr/local/rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
/usr/local/rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/timeout.rb:101:in `call'
/usr/local/rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/timeout.rb:101:in `timeout'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/channel.rb:1175:in `exchange_declare'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/exchange.rb:245:in `declare!'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/exchange.rb:83:in `initialize'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/channel.rb:344:in `new'
/home/deployer/project/shared/bundle/ruby/2.1.0/bundler/gems/bunni-cd347c9da757/lib/bunni/channel.rb:344:in `topic'
/home/deployer/project/releases/20151005085039/app/services/publisher.rb:32:in `publish'

If create connection and channel in Publisher directly than works.

class Publisher
  ...
  $rabbitmq_connection = Bunni.new "amqp://#{user}:#{pass}@#{host}:#{port}#{vhost}"
  $rabbitmq_connection.start
  $rabbitmq_channel = $rabbitmq_connection.create_channel
  ...
end

Puma settings

#config/deploy.rb
set :puma_workers, 4
set :puma_threads, [4, 16]
set :puma_init_active_record, true
set :puma_bind, %w(tcp://0.0.0.0:9291 tcp://0.0.0.0:9292 unix:///home/deployer/project/current/tmp/sockets/puma.sock)

How should I initialize connection and create channel in my case? Thanks

Breed

hi, I have run into a similar problem, have you been able to solve the issue? – Bannasch

I have run into this as well, any update on this? – Gazette

I have the exact same issue, just to recap: a) global connection established in initializer b) global conn used from publisher c) connection fetched directly in publisher works, global connection fails to create_channel (get Timeout error exactly the same as OP) – Edgebone

Answer 1

If it's a threading issue (considering Puma is a multi-threaded environment) creating a new RabbitMQ connection for each publish action in your Publisher class may help. Could be caused by threads sharing one connection.

class Publisher
  def self.publish
    rabbitmq_connection = Bunny.new("amqp://#{user}:#{pass}@#{host}:#{port}#{vhost}")
    rabbitmq_connection.start
    rabbitmq_channel = rabbitmq_connection.create_channel

    # ...

    rabbitmq_channel.close
    rabbitmq_connection.close
  end
end

Ashantiashbaugh

Hi, thanks for answer Matthew but the idea is to not have to create a new connection each time if possible – Edgebone

Answer 2

This happens when puma is configured to run in clustered mode, when you have multiple workers:

# config/puma.rb

max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 2 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count
worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development"
port ENV.fetch("PORT") { 3000 }
environment ENV.fetch("RAILS_ENV") { "development" }
pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }
preload_app!
plugin :tmp_restart

workers ENV.fetch("WEB_CONCURRENCY") { 2 }

$rabbitmq_connection = Bunny.new
$rabbitmq_connection.start
$rabbitmq_channel = $rabbitmq_connection.create_channel

With multiple workers this will time out:

class HomeController < ApplicationController
  def show
    exchange = $rabbitmq_channel.topic("some", auto_delete: true)
  end
end

Puma will fork the main process to create workers, which apparently has some issues:

The fork(2) system call has several gotchas associated with it:

• Unintentional file descriptor sharing
• The fact that a forked child process only inherits one thread and therefore the network I/O thread is not inherited

To avoid both problems, connect to RabbitMQ after the master process forks workers

"network I/O thread is not inherited" - I'm guessing this is why connection times out.

https://github.com/ruby-amqp/bunny/blob/main/docs/guides/connecting.md#connecting-in-web-applications-ruby-on-rails-sinatra-etc

---

The Fix

Create connection after the fork:

# config/puma.rb

on_worker_boot do
  $rabbitmq_connection = Bunny.new
  $rabbitmq_connection.start
  $rabbitmq_channel = $rabbitmq_connection.create_channel
end

https://www.rubydoc.info/gems/puma/Puma/DSL#on_worker_boot-instance_method

Greatnephew

It looks promising, but workers ENV.fetch("WEB_CONCURRENCY") { 2 } isn't enabled in my dev env where this is happening. I tried enabling and adding the on_worker_boot code, but $rabbitmq_connection behaves strangely when called from the publisher, I can initialize it from binding.pry but I get Exception error="undefined method create_channel' for nil:NilClass"` from Sneakers when trying to actually use it – Edgebone

@Edgebone probably should've mentioned Sneakers before. which has it's own hooks like after_fork where you can set up your connection github.com/jondot/sneakers/wiki/Forking#forking-hooks – Greatnephew