Rails stops responding if Idle

Question

I've been having this problem for a while and searched a lot about this, but nothing seems to solve my problem.

I have an application running in Ruby on Rails with NGINX and UNICORN everything with Docker.

If I run the application (docker-compose up) the web site runs perfectly for a few minutes. If the web site stays idle for about 5 to 10 minutes, and I send another request the rails application stops responding giving time out from Unicorn.

The weird part is that every application I develop, have the same problem and the only thing in common these applications have are the Gemfile.

What I found so far is that when the request is sent, NGINX receives this request, pass it to Unicorn, Unicorn receives it and passes to ruby... which doesn't answer and then Unicorn gives me the 502 bad request.

I'm really lost at this one.

this is my NGINX config file:

upstream applicationName {
    # Path to Puma SOCK file, as defined previously
    server unix:/tmp/applicationName.sock fail_timeout=0;
}

server {
  listen 80;
  server_name dev.applicationName.com.br;

  location / {
    autoindex on;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Host $host;
    # time out settings
    proxy_connect_timeout 159s;
    proxy_send_timeout   600;
    proxy_read_timeout   600;
    proxy_buffer_size    64k;
    proxy_buffers     16 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass_header Set-Cookie;
    proxy_redirect     off;
    proxy_hide_header  Vary;
    proxy_set_header   Accept-Encoding '';
    proxy_ignore_headers Cache-Control Expires;
    proxy_set_header   Referer $http_referer;
    proxy_set_header   Host   $host;
    proxy_set_header   Cookie $http_cookie;
    proxy_set_header   X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://applicationName;
  }
}

And this is my unicorn config

@dir = File.expand_path(File.dirname(__FILE__)) + "/.."

worker_processes 2
working_directory @dir

timeout 10

listen File.join('/tmp/applicationName.sock')

preload_app true# if ENV['RAILS_ENV'] != 'development'

GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

check_client_connection false

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

This is the Gemfile (nothing wrong with it)

source 'https://rubygems.org'
gem 'rails', '4.2.4'
gem 'unicorn-rails', '~> 2.2'
gem 'pg'
gem 'mysql2', '~> 0.3.18'
gem 'sass-rails', '~> 5.0'
gem 'uglifier', '>= 1.3.0'
gem 'coffee-rails', '~> 4.1.0'
gem 'duktape'
gem 'jquery-rails'
gem 'turbolinks'
gem 'jbuilder', '~> 2.0'
gem 'bootstrap-sass'
gem 'devise'
gem 'simple_form'
gem 'minitest'
gem "paperclip", "~> 4.3"
gem 'aws-sdk', '< 2.0'
gem 'mail_form', '~> 1.5.0.rc'
gem 'sendgrid-ruby'
gem 'zopim_rails'
gem 'meta-tags'
gem 'ckeditor'
gem 'slick_rails'

group :development do
  gem 'better_errors'
  gem 'binding_of_caller', :platforms=>[:mri_20]
  gem 'quiet_assets'
  gem 'rails_layout'
  gem 'spring-commands-rspec'
  gem 'web-console', '~> 2.0'
  gem 'spring'
end
group :production do
  gem 'therubyracer'
end
group :development, :test do
  gem 'factory_girl_rails'
  gem 'faker'
  gem 'pry-rails'
  gem 'pry-rescue'
  gem 'rspec-rails'
  gem 'rubocop'
  gem 'byebug'
end

group :test do
  gem 'capybara'
  gem 'database_cleaner'
  gem 'launchy'
  gem 'selenium-webdriver'
end

The log on the server shows me this, when the error happens:

nginx_1 | 172.17.0.1 - - [05/Feb/2016:12:52:41 +0000] "GET /products HTTP/1.1" 502 574 "http://dev.nutrimais.com.br/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36"
web_1   | E, [2016-02-05T12:52:41.551728 #1] ERROR -- : reaped #<Process::Status: pid 9 SIGKILL (signal 9)> worker=0

And on the development log:

Started GET "/products" for 127.0.0.1 at 2016-02-05 12:52:18 +0000
Cannot render console from 127.0.0.1! Allowed networks: 127.0.0.1, ::1, 127.0.0.0/127.255.255.255

Delanty

Have you tried to debug it a little bit more? Like launching your rails server directly (rails s production) and accessing that or trying an equivalent of unicorn like passenger or puma? If things still go wrong with alternatives, then that would definitely narrow things down, don't you think? :) – Gabo

I've tried Puma and it was worse. The application started getting slow after a few minutes and crashing, making the user to enter the web site again. – Delanty

That is strange... And what happens when you launch "rails server production" and you access it directly through your_server_ip:3000 ? – Gabo

As the project was working with docker, I'm now trying to make it run directly – Delanty

the error in the rails log is from web console. try to add the ip to the whitelist and see what hppns – Decrepitude

Already did that and it makes the log go away, but still doesn't solve it – Delanty

Possible duplicate of Server Ruby on Rails + unicorn + nginx stop responding – Divisible

Answer 1

In deed it's a problem with the firewall of my Linux server and the DB, which is located in another server. What I did is bring the DB server to the same machine as my rails application. Another way to fix it is change the firewall rules ofthe server machine. I haven't done that and not sure how to. It looks like this problem only happens with MySQL.

Delanty

Answer 2

Your fix of moving the MySQL to the same server also fixed the issue for me but that's not suitable for my production environment so I dug deeper into what made the difference with that setup.

Turned out it was nothing related to the application stack or configuration, but a NAT timeout that was killing my sessions.

I had my server on Azure and my database in AWS. Azure has a 4 minute unconfigurable NAT timeout on outbound connections. That means if a database connection was idle for more than 4 minutes, Azure would just silently kill the port mapping and any traffic the Rails app tried to send down that port silently went into a black hole.

The fix was per this blob at the OS level on the Rails server: Drop the TCP keepalive interval to half the NAT timeout, and adjust the keepalive interval and retransmissions to appropriate values for the new keepalive interval.

For Linux, you should change these kernel variables as follows:

sudo sysctl net.ipv4.tcp_keepalive_time = 120     
sudo sysctl net.ipv4.tcp_keepalive_intvl = 30 
sudo sysctl net.ipv4.tcp_keepalive_probes = 8

For Windows, you should change the registry values below.

All DWORDs in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters:

KeepAliveInterval = 30 
KeepAliveTime = 120 
TcpMaxDataRetransmissions = 8

Buckner

Answer 3

I have heard about such issues with unicorn that if a site is not accessed for around (average) 30 minutes the next query will timeout, and it will timeout on all the workers opened. I am not sure why this happens. I hope the folowing link might help you

http://bogomips.org/unicorn-public/[e-mail]/t/

Weiweibel

reading your link, looks like it's a problem with the firewall on linux with the DB connection. But is it the linux of the docker image or the linux of my server? And the answer is not clear of what I have to do. – Delanty