Cucumber tests suddenly stops

Question

I have feature like this:

Feature: Searching chats
In order to find chats
As an user
I want to find different chats by username or ad name

Background:
  Given System prepares for chats
  And There is a few machines with names:
    | machine_1 |
    | machine_2 |
    | machine_3 |
  And There is a few services with names:
    | service_1 |
    | service_2 |
    | service_3 |
  And I have chats with ads owners

Scenario: Searching chats
  When I am logged in as a "user"
  And I go to chats page   # <- stops here
  Then I should see search results when I fill form with:
    | input  | results |
    | ma     | machine_1, machine_2, machine_3 |
    | se     | service_1, service_2, service_3 |                                                                         

When I start cucumber feature (or scenario), it suddenly stops at step "And I go to chats page" without any error message. Result looks like:

[alex@MacBookPro ~/my_project | master]$ cucumber features/chat/search.feature
Using the default profile...
@javascript
Feature: Searching chats
  In order to find chats
  As an user
  I want to find different chats by username or ad name

  Background:                               # features/chat/search.feature:8
    Given System prepares for chats         # features/step_definitions/chats.steps.rb:11
    And There is a few machines with names: # features/step_definitions/machine.steps.rb:10
      | machine_1 |
      | machine_2 |
      | machine_3 |
    And There is a few services with names: # features/step_definitions/service.steps.rb:144
      | service_1 |
      | service_2 |
      | service_3 |
    And I have chats with ads owners        # features/step_definitions/chats.steps.rb:5

  Scenario: Searching chats                                 # features/chat/search.feature:20
    When I am logged in as a "user"                         # features/step_definitions/user.steps.rb:68
    And I go to chats page                                  # features/step_definitions/chats.steps.rb:17
[alex@MacBookPro~/my_project | master]$

That's my "falling" steps:

When /^I go to chats page$/ do
  visit root_path

  within('.global-menu') do
    click_on username(@current_user)
    click_on I18n.t('views.menu.profile.links.dashboard')
  end

  click_on I18n.t('views.menu.profile.links.chats')
end

Then(/^I should see search results when I fill form with:$/) do |table|
  table.hashes.each do |search_data|
    ### searching ###
    @page.query.set search_data['input']

    # for AJAX search
    sleep 1

    ### show results ###
    search_data['results'].split(', ').each do |res|
      page.should have_content res.mb_chars.upcase
    end

    within('#chats') do
      page.all('.chat').length.should == search_data['results'].split(', ').size
    end
  end
end

I'm using capybara-webkit with cucumber. That's my env.rb:

require 'rubygems'
require 'spork'
require 'capybara'
require 'capybara/rspec'
require 'selenium-webdriver'
require 'site_prism'
require 'capybara-screenshot/cucumber'
# require 'cucumber/rails'


# 1) Tag your scenario (or feature) with @allow-rescue
#
# 2) Set the value below to true. Beware that doing this globally is not
# recommended as it will mask a lot of errors for you!
#
# ActionController::Base.allow_rescue = false

#############################################################################

ENV['SKIP_RAILS_ADMIN_INITIALIZER'] = 'false'                                   # This fixes weird errors with cucumber + rails_admin (http://makandracards.com/makandra/9597-rake-spec-+-rails_admin-weirdly-failing-specs).

#############################################################################

Before do
  DatabaseCleaner.strategy = :truncation

  DatabaseCleaner.clean
  FactoryGirl.create(:setting)
  ContactType.generate_contact_types
  ContactType.generate_ims
end

Spork.prefork do
  require 'cucumber/rails'
  require 'email_spec' # add this line if you use spork
  require 'email_spec/cucumber'
  Capybara.default_selector = :css
end

Spork.each_run do
  ActionController::Base.allow_rescue = false

  begin
    DatabaseCleaner.strategy = :truncation
  rescue NameError
    raise "You need to add database_cleaner to your Gemfile (in the :test group) if you wish to use it."
  end

  Capybara.register_driver :webkit do |app|
    Capybara::Webkit::Driver.new(app, :stderr => nil)
  end

  Capybara.javascript_driver = :webkit

  Cucumber::Rails::Database.javascript_strategy = :truncation
end

Problem appeard when I have updaded my project to rails4. Any ideas?

Camfort

I said there is no error. Updated my question. – Camfort

show your step And I go to chats page – Paola

I have the same error: tests worked in Rails 3 and they stopped to work in Rails 4 (it looks like it terminated during the test). – Jinja

I guess you shall to debug it with pry insert require 'pry'; binding.pry – Paola

I have a similar issue: I see that the console output stops, but the tests keep running (my firefox keeps moving). Only I have no idea afterwards if my tests were succesful or not (since there is no scripts output). It seems as if somehow opening selenium changes the STDOUT or something. But, logging the console output to a file does not improve it at all. – Daukas

@Daukas If I run selenium instead of webkit, then i see that tests kinda pass in browser, however they are still terminating in the console. Did you find any workaround? I moved to the Poltergeist instead of webkit and number of terminating tests decreased but still i have some of them (i just commented them). – Jinja

+1 i'm also experiencing this only not with Cucumber, just rspec. The tests seem to go through, but there is something weird happening to the output. What's strange for me is that this occurs only a certain pages. My app has a very asynchronous architecture in which a page can load any number of pagelets(similar to facebook). Its on these page that load more than one pagelet(ie more than 1 ajax request) that I experience this drop-out issue in my tests. As well, tests were passing in rails3, and not in rails 4.1. Seems like some sort of threading/race condition issue to me. – Inoperable

As well, switching js drivers(webkit/webkit-debug/selenium-chrome/poltergeist) does not seem to help. I'd love to see if anyone has any real solutions to this. – Inoperable

Answer 1

A lot of the developers I managed find Capybara-webkit to be really problematic and inconsistent.

poltergeist/PhantomJS has a lot of advantages over it. Generally it:

• is more deterministic, in that problem scenarios are more likely to fail consistently than to be flaky
• is less machine-dependent; our suite now behaves pretty much the same in all of our test environments
• gives better error messages
• fails when there are Javascript errors, even if the test would otherwise pass
• doesn’t hang, and
• is easier to install.

Here's a good post from Dave Schweisguth about his presentation at the the February Automated Testing SF meetup, where he discussed his company (Fandor)'s testing setup/environment, issues, and troubleshooting and a quick comparison. It might help you track down your problem.

Tailorbird

Answer 2

Ok, I don't have an answer, but I have more evidence that leads to a workaround.

This applies to rspec, but I assume it should be the same for Cucumber as well:

rspec spec/ --formatter progress --out rspec.output.txt

It looks like the pointer to STDOUT is getting mashed somehow. By specifying an output file and tailing it, you should see the full output.

I tried all the different formatters and no matter what, if they output to STDOUT, the output gets lost somewhere along the way.

Inoperable

Also, I've filed this issue with rspec-core: github.com/rspec/rspec-core/issues/1520#issuecomment-42490138 See the comment where you can see the File#stat call with different inode/uid/gid/rdev attributes – Inoperable

Answer 3

I've had a similar issue, for unknown reasons when using Selenium web driver. But when I've switched to Poltergeist (PhantomJS) it started to work.

Also, I noticed, that you are requiring selenium driver, but then, you are using the Webkit.

And after changing the driver, try running everything without a Spork running.

Boyar

I used selenium before webkit. Now I have removed it completely, started without spork, and tests still terminate. – Camfort

@Camfort Have you tried another driver, like f.i. poltergeist? – Boyar

oh sorry, forgot about that. Yep, this is helped, but now I got problem in another scenario with same result. – Camfort

Answer 4

Use thin web server instead of webkit and put the following code in features/support/env.rb:

Capybara.server do |app, port|
 require 'rack/handler/thin'
 Rack::Handler::Thin.run(app, :Port => port)
end

Read more about this solution using thin server from the following link:

Solution of same problem using thin web server, and read this solution same solution by another one.

Ruggiero