PG::UnableToSend: no connection to the server in Rails

Question

I have a production server running ubuntu 14.04, Rails 4.2.0, postgresql 9.6.1 with gem pg 0.21.0/0.20.0. In last few days, there is constantly error with accessing to a table customer_input_datax_records in psql server.

D, [2017-07-20T18:08:39.166897 #1244] DEBUG -- :   CustomerInputDatax::Record Load (0.1ms)  SELECT "customer_input_datax_records".* FROM "customer_input_datax_records" WHERE ("customer_input_datax_records"."status" != $1)  [["status", "email_sent"]]
E, [2017-07-20T18:08:39.166990 #1244] ERROR -- : PG::UnableToSend: no connection to the server
: SELECT "customer_input_datax_records".* FROM "customer_input_datax_records" WHERE ("customer_input_datax_records"."status" != $1)

The code which call to access the db server is with Rufus scheduler 3.4.2 loop:

s = Rufus::Scheduler.singleton
s.every '2m' do

    new_signups = CustomerInputDatax::Record.where.not(:status => 'email_sent').all

.......
end

After restart the server, usually there is with first request (or a few). But after some time (ex, 1 or 2 hours), the issue starts to show up. But the app seems running fine (accessing records with read/write & creating new). There are some online posts about the error. However the problem seems not the one I am having. Before I re-install the psql server, I would like to get some ideas about what causes the no connection.

UPDATE: database.yml

production:
  adapter: postgresql
  encoding: unicode
  host: localhost
  database: wb_production
  pool: 5
  username: postgres
  password: xxxxxxx

Nieman

Are you having this issue in development or production? Are your local connections to the db trusted? – Semibreve

Could you add to your question the database.yml for your environment? Are you sure that your pg_hba.conf is correctly set up? – Semibreve

It is a on production server. The server has been running for over a year and the problem is recent. There is no change on the config/setup. – Nieman

The app is deployed on an AWS service? – Semibreve

Could it be related to some gems you are using? I saw some issues on [delayed_job](github.com/collectiveidea/delayed_job/issues/628], parallel ... – Semibreve

It is on Ali cloud. The error is specifically caused by a line of code. Just updated the post. new_signups = CustomerInputDatax::Record.where.not(:status => 'email_sent').all – Nieman

Actually I don't know Ali cloud. Did you try to use another scheduler like gem whenever and/or to run this command invoking a rake task instead of calling directly the activerecord model? – Semibreve

Downgraded rufus_scheduler from 3.4.2 to 3.3.4 and the system starts to access the table fine. It seems to be related to the newer version of rufus_scheduler. Will check out whenever. thank you. – Nieman

Answer 1

So, the error is "RAILS: PG::UnableToSend: no connection to the server".

That reminds me of Connection pool issue with ActiveRecord objects in rufus-scheduler

You could do

s = Rufus::Scheduler.singleton
s.every '2m' do

  ActiveRecord::Base.connection_pool.with_connection do
    new_signups = CustomerInputDatax::Record
      .where.not(status: 'email_sent')
      .all
    # ...
  end
end

digging

It would be great to know more about the problem.

I'd suggest trying this code:

s = Rufus::Scheduler.singleton

def s.on_error(job, error)

  Rails.logger.error(
    "err#{error.object_id} rufus-scheduler intercepted #{error.inspect}" +
    " in job #{job.inspect}")
  error.backtrace.each_with_index do |line, i|
    Rails.logger.error(
      "err#{error.object_id} #{i}: #{line}")
  end
end

s.every '2m' do
  new_signups = CustomerInputDatax::Record.where.not(:status => 'email_sent').all
  # .......
end

As soon as the problem manifests itself, I'd look for the on_error full output in the Rails log.

This on_error comes from https://github.com/jmettraux/rufus-scheduler#rufusscheduleron_errorjob-error

Nd

Answer 2

As we discuss in the comments, the problem seems related to your rufus version. I would suggest you to check out whenever gem and to invoke a rake task instead of calling directly the activerecord model.

It could be a good idea, however, to open an issue with the traceback of your error in the rufus-scheduler repo on github (just to let then know...)

Semibreve