Is there a good way to debug order dependent test failures in RSpec (RSpec2)?

Question

Too often people write tests that don't clean up after themselves when they mess with state. Often this doesn't matter since objects tend to be torn down and recreated for most tests, but there are some unfortunate cases where there's global state on objects that persist for the entire test run, and when you run tests, that depend on and modify that global state, in a certain order, they fail.

These tests and possibly implementations obviously need to be fixed, but it's a pain to try to figure out what's causing the failure when the tests that affect each other may not be the only things in the full test suite. It's especially difficult when it's not initially clear that the failures are order dependent, and may fail intermittently or on one machine but not another. For example:

rspec test1_spec.rb test2_spec.rb # failures in test2
rspec test2_spec.rb test1_spec.rb # no failures

In RSpec 1 there were some options (--reverse, --loadby) for ordering test runs, but those have disappeared in RSpec 2 and were only minimally helpful in debugging these issues anyway.

I'm not sure of the ordering that either RSpec 1 or RSpec 2 use by default, but one custom designed test suite I used in the past randomly ordered the tests on every run so that these failures came to light more quickly. In the test output the seed that was used to determine ordering was printed with the results so that it was easy to reproduce the failures even if you had to do some work to narrow down the individual tests in the suite that were causing them. There were then options that allowed you to start and stop at any given test file in the order, which allowed you to easily do a binary search to find the problem tests.

I have not found any such utilities in RSpec, so I'm asking here: What are some good ways people have found to debug these types of order dependent test failures?

Laureenlaurel

This is very rhetorical. Can you try simplifying the question to be more to the point? – Mimamsa

That's what the headline/title is for. And the very last sentence. The rest is to clarify what I mean by "order dependent test failure" since I didn't find other questions that dealt with that, and then trying to describe what sort of solution I'm looking for. If both of those are obvious to others reading the question then I agree the other verbage is overly rhetorical, but I'm not sure it's that obvious. – Laureenlaurel

Answer 1

Found my own question 4 years later, and now rspec has a --order flag that lets you set random order, and if you get order dependent failures reproduce the order with --seed 123 where the seed is printed out on every spec run.

https://www.relishapp.com/rspec/rspec-core/v/2-13/docs/command-line/order-new-in-rspec-core-2-8

Laureenlaurel

Answer 2

There is now a --bisect flag that will find the minimum set of tests to run to reproduce the failure. Try:

$ rspec --bisect=verbose

It might also be useful to use the --fail-fast flag with it.

Pennipennie

This is an amazing feature. Solved my problem perfectly. Thanks! Should be top answer. – Docia

Answer 3

I wouldn't say I have a good answer, and I'd love to here some better solutions than mine. That said...

The only real technique I have for debugging these issues is adding a global (via spec_helper) hook for printing some aspect of database state (my usual culprit) before and after each test (conditioned to check if I care or not). A recent example was adding something like this to my spec_helper.rb.

Spec::Runner.configure do |config|
  config.before(:each) do
    $label_count = Label.count
  end

  config.after(:each) do
    label_diff = Label.count - $label_count
    $label_count = Label.count
    puts "#{self.class.description} #{description} altered label count by #{label_diff}" if label_diff != 0
  end
end

Insistency

Thanks for the answer, but it doesn't help me since the software I'm testing doesn't have a database, and the order dependent failures aren't so predictable that a print statement this general would help me :-( – Laureenlaurel

Answer 4

We have a single test in our Continuous Integration setup that globs the spec/ directory of a Rails app and runs each of them against each other.

Takes a lot of time but we found 5 or 6 dependencies that way.

Fleta

Do you have the code for that handy? It sounds rather useful. – Paediatrician

Answer 5

Here is some quick dirty script I wrote to debug order-dependent failure - https://gist.github.com/biomancer/ddf59bd841dbf0c448f7

It consists of 2 parts.

First one is intended to run rspec suit multiple times with different seed and dump results to rspec_[ok|fail]_[seed].txt files in current directory to gather stats.

The second part is iterating through all these files, extracts test group names and analyzes their position to the affected test to make assumptions about dependencies and forms some 'risk' groups - safe, unsafe, etc. The script output explains other details and group meanings.

This script will work correctly only for simple dependencies and only if the affected test is failing for some seeds and passes for another ones, but I think it's still better than nothing. In my case it was complex dependency when effect could be cancelled by another test but this script helped me to get directions after running its analyze part multiple times on different sets of dumps, specifically only on the failed ones (I just moved 'ok' dumps out of current directory).

Merrymerryandrew

Answer 6

Its most likely some state persisting between tests so make sure your database and any other data stores (include class var's and globals) are reset after every test. The database_cleaner gem might help.

Robichaux

Answer 7

Rspec Search and Destroy

is meant to help with this problem.

https://github.com/shepmaster/rspec-search-and-destroy

Visconti

This gem seems abandoned and doesn't work with rspec3 – Enteric