deadlock detected with capybara-webkit

Question

I'm trying to pass this spec :

scenario "Edit a service", js: true do
  service = create_service_for(provider, title: "First service")
  fill_edit_service_form(service)
  expect(page).to have_css('#price', text: '10,00 $')
end

This is a standard rails spec using capybara. I am using capybara-webkit for all scenario with javascript. When I'm trying to pass it, sometimes it works, sometimes it mark than there is a missing record in the database and sometime I have this error :

Run options: include {:locations=>{"./spec/acceptances/provider_services_spec.rb"=>[31]}}
[K  1) Provider Services Edit a service
     Failure/Error: Unable to find matching line from backtrace
     ActiveRecord::StatementInvalid:
       PG::TRDeadlockDetected: ERROR:  deadlock detected
       DETAIL:  Process 24164 waits for AccessExclusiveLock on relation 3446991 of database 3446538; blocked by process 24184.
       Process 24184 waits for AccessShareLock on relation 3446902 of database 3446538; blocked by process 24164.
       HINT:  See server log for query details.
       : ALTER TABLE "active_admin_comments" DISABLE TRIGGER ALL;ALTER TABLE "provider_service_territory_provideds" DISABLE TRIGGER ALL;ALTER TABLE "provider_services" DISABLE TRIGGER ALL;ALTER TABLE "provider_divisions" DISABLE TRIGGER ALL;ALTER TABLE "provider_profiles" DISABLE TRIGGER ALL;ALTER TABLE "provider_service_intervention_level_provideds" DISABLE TRIGGER ALL;ALTER TABLE "provider_service_medium_provideds" DISABLE TRIGGER ALL;ALTER TABLE "provider_service_service_provideds" DISABLE TRIGGER ALL;ALTER TABLE "regions" DISABLE TRIGGER ALL;ALTER TABLE "service_formulas" DISABLE TRIGGER ALL;ALTER TABLE "region_translations" DISABLE TRIGGER ALL;ALTER TABLE "artists" DISABLE TRIGGER ALL;ALTER TABLE "admin_users" DISABLE TRIGGER ALL;ALTER TABLE "schema_migrations" DISABLE TRIGGER ALL;ALTER TABLE "services" DISABLE TRIGGER ALL;ALTER TABLE "services_provided" DISABLE TRIGGER ALL;ALTER TABLE "territories_provided" DISABLE TRIGGER ALL;ALTER TABLE "users" DISABLE TRIGGER ALL;ALTER TABLE "countries" DISABLE TRIGGER ALL;ALTER TABLE "coupons" DISABLE TRIGGER ALL;ALTER TABLE "currencies" DISABLE TRIGGER ALL;ALTER TABLE "formulas" DISABLE TRIGGER ALL;ALTER TABLE "collector_profiles" DISABLE TRIGGER ALL;ALTER TABLE "country_translations" DISABLE TRIGGER ALL;ALTER TABLE "images" DISABLE TRIGGER ALL;ALTER TABLE "intervention_levels_provided" DISABLE TRIGGER ALL;ALTER TABLE "measure_units" DISABLE TRIGGER ALL;ALTER TABLE "media_provided" DISABLE TRIGGER ALL;ALTER TABLE "messages" DISABLE TRIGGER ALL;ALTER TABLE "page_part_translations" DISABLE TRIGGER ALL;ALTER TABLE "orders" DISABLE TRIGGER ALL;ALTER TABLE "painting_categories" DISABLE TRIGGER ALL;ALTER TABLE "page_services" DISABLE TRIGGER ALL;ALTER TABLE "page_translations" DISABLE TRIGGER ALL;ALTER TABLE "painting_category_translations" DISABLE TRIGGER ALL;ALTER TABLE "page_parts" DISABLE TRIGGER ALL;ALTER TABLE "pages" DISABLE TRIGGER ALL;ALTER TABLE "painting_provenances" DISABLE TRIGGER ALL;ALTER TABLE "painting_prices" DISABLE TRIGGER ALL;ALTER TABLE "painting_technic_translations" DISABLE TRIGGER ALL;ALTER TABLE "painting_technics" DISABLE TRIGGER ALL;ALTER TABLE "painting_type_translations" DISABLE TRIGGER ALL;ALTER TABLE "period_translations" DISABLE TRIGGER ALL;ALTER TABLE "painting_types" DISABLE TRIGGER ALL;ALTER TABLE "paintings" DISABLE TRIGGER ALL;ALTER TABLE "periods" DISABLE TRIGGER ALL
     # -e:1:in `<main>'

 1/1 |========================= 100 ==========================>| Time: 00:00:03 

Finished in 3.41 seconds
1 example, 1 failure

Failed examples:

rspec ./spec/acceptances/provider_services_spec.rb:29 # Provider Services Edit a service
     Screenshot: /home/dougui/rails/lescollectionneurs/tmp/capybara/screenshot_2014-02-20-18-22-55.658.png

Randomized with seed 34053

I have a lock in a table. This not always the same table.

It's better when I do this :

Capybara.reset_sessions!
DatabaseCleaner.clean

Before and after the spec but it is not always working. If I run all my specs in the same file, I does not work.

It appends when I was working on a unrelated things. It works with poltergeist.

This is my spec_helper file :

ENV["RAILS_ENV"] ||= 'test'
require 'rubygems'
require File.expand_path("../../config/environment", __FILE__)
require 'rspec/rails'
require 'rspec/autorun'
require 'capybara/rails'
require 'capybara-webkit'
require 'database_cleaner'
require 'capybara/firebug'
require 'capybara-screenshot/rspec'

if defined?(Spring)
  Spring.watch "#{Rails.root}/spec/factories"
else
  require 'shoulda-matchers'
end

Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

RSpec.configure do |config|
  config.infer_base_class_for_anonymous_controllers = false
  config.use_transactional_fixtures = false
  config.order = "random"

  config.include FactoryGirl::Syntax::Methods
  config.include Warden::Test::Helpers, type: :feature
  config.include FeatureHelpers, type: :feature
  config.include Devise::TestHelpers, type: :controller
  config.include ControllerHelpers, type: :controller
  config.include EmailSpec::Helpers
  config.include EmailSpec::Matchers

  config.before(:suite) do
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    if example.metadata[:js]
      DatabaseCleaner.strategy = :truncation
    else
      DatabaseCleaner.start
    end
  end

  config.after(:each) do
    DatabaseCleaner.clean
    DatabaseCleaner.strategy = :transaction if example.metadata[:js]
  end
end

Capybara.javascript_driver = :webkit

include ActionDispatch::TestProcess
I18n.locale = :fr

Geocoder.configure(:lookup => :test)

Geocoder::Lookup::Test.set_default_stub(
  [
    {
      'latitude'     => 40.7143528,
      'longitude'    => -74.0059731,
      'address'      => 'New York, NY, USA',
      'state'        => 'New York',
      'state_code'   => 'NY',
      'country'      => 'United States',
      'country_code' => 'US'
    }
  ]
)

Here are the gem used : https://github.com/GCorbel/lescollectionneursassocies/blob/master/Gemfile.

Any suggestions?

For more details, please look at the Github repo : https://github.com/GCorbel/lescollectionneursassocies/.

Trangtranquada

Do you run tests with parallel_tests gem? – Natterjack

Answer 1

This is happening because you have two threads, the test and the request, modifying the database asynchronously. It looks like you've rightly been experimenting with different configurations in your spec_helper. I've spend a fair amount of time struggling with this same issue, and have come up with this:

# adapted from https://gist.github.com/moonfly/4950750

require 'database_cleaner'

RSpec.configure do |config|

  config.use_transactional_fixtures = false

  config.before( :suite ) do
    DatabaseCleaner.clean_with :truncation
    DatabaseCleaner.strategy = :transaction
  end

  config.around( :each ) do |spec|
    if spec.metadata[:js]
      # JS => doesn't share connections => can't use transactions
      spec.run
      DatabaseCleaner.clean_with :deletion
    else
      # No JS/Devise => run with Rack::Test => transactions are ok
      DatabaseCleaner.start
      spec.run
      DatabaseCleaner.clean

      # see https://github.com/bmabey/database_cleaner/issues/99
      begin
        ActiveRecord::Base.connection.send :rollback_transaction_records, true
      rescue
      end
    end
  end

end

I keep it all in support/database_cleaner_configuration.rb, which would work for you, as well, or you can simply replace what you've got in your spec_helper. If this doesn't work, let me know - I have a few other ideas, but they're kookier, and not worth getting into if this works, which it probably will...

Note:

It might be worth mentioning that Rails 5.1+ solves the database problem. Eileen Uchitelle on the Rails team made the changes necessary to run ensure test threads and the Rails server can run in the same process by sharing the database connection.

Bister

Also, maybe check this out - #8965037 – Bister

This seems to be working so thanks. I did need to change the strategy to DatabaseCleaner.strategy = :truncation as per the comment for when spec.metadata[:js] however. – Companionway

It might be worth mentioning that Rails 5.1+ solves the database problem. Eileen Uchitelle on the Rails team made the changes necessary to run ensure test threads and the Rails server can run in the same process by sharing the database connection. github.com/rails/rails/pull/28083 – Bister

Answer 2

You have the deadlock when there are two threads which run the tests. For example open a tab in your terminal and run tests with the rspec command. Immediately open second tab in the terminal and run the tests there too with the same command.

So, I guess you just have run all tests in one tab and trying to run particular test with the rspec spec/acceptances/provider_services_spec.rb:31 command in other tab.

Natterjack

No, it is not that. I do a loop to run the test 10 times and I did nothing until end. 8 times, it works, 2 times, it failed. I watched all processes during this time and when it failed there was only one process with ruby and one with webkit_server. Any other idea? – Trangtranquada

Do you run test with parallel_tests? – Natterjack

Answer 3

The most probable reason is there must be some query going on through ajax / database cleaner is truncating the database prematurely while sql query is running. One of the method is to make webkit driver wait till the ajax request is finished. Add this method in the test helpers and call it after clicking the button / submitting the ajax form.

    def wait_for_ajax_to_finish
    looping = true
    loop do
      sleep 1
      begin
        count = page.evaluate_script('window.running_ajax_calls').to_i
        looping = false if count == 0
      rescue => e
        if e.message.include? 'HTMLunitCorejsJavascript::Undefined'
          raise "For 'wait for the AJAX call to finish' to work, please include culerity.js after including jQuery."
        else
          raise e
        end
      end
    end
  end

Frisbie

Answer 4

I was having different tests fail randomly (like 1 test out of 500, every other time I ran the suite).

This may not be the prettiest solution, but I solved the problem by just rescuing and retrying the database cleaner after 2 seconds, like so:

config.after(:each) do |example|
  begin
    DatabaseCleaner.clean
  rescue Exception => e
    # Recover from thread locks and retry the database clean after a slight delay
    sleep 2
    DatabaseCleaner.clean
  end
end

Chiropteran