Why do Test::Unit testcases start up so slowly?

Question

>rails -v
Rails 1.2.6

>ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]

When I run a test fixture (that tests a rails model class) like this, it takes 20-30 secs to start executing these tests (show the "Loaded suite..."). What gives?

>ruby test\unit\category_test.rb

require File.dirname(__FILE__) + '/../test_helper'

class CategoryTest < Test::Unit::TestCase
    def setup
        Category.delete_all
    end
    def test_create
        obCategoryEntry = Category.new({:name=>'Apparel'})
        assert obCategoryEntry.save, obCategoryEntry.errors.full_messages.join(', ')
        assert_equal 1, Category.count

        assert_not_nil Category.find(:all, :conditions=>"name='Apparel'")
    end
    #.. 1 more test here
end

This one is Rails using a MySql DB with no fixtures. This time it clocked 30secs+ to startup.

Shandeigh

in test_helper.rb add "self.use_transactional_fixtures = true" and you can drop the delete_all in the set up. I know you are not using fixtures but this line will ensure that the DB is set to the original state after each test run. Note that it won't delete data that is already in there. – Martyr

Are you using windows? I have seen it take 30 seconds to load the rails env under windows, where the same project would like 2 seconds on linux. – Polyvinyl

Yes. indeed.. see the ruby version output above.. i386-mswin32. – Shandeigh

Answer 1

Take a look at this Rails Test Server.

A quote from the author:

"Every time you run a test in a Rails application, the whole environment is loaded, including libraries that don’t change between two consecutive runs. That can take a considerable amount of time. What if we could load the environment once, and only reload the changing parts before each run? Introducing RailsTestServing.

With RailsTestServing, the run time of a single test file has gone from 8 seconds down to .2 of a second on my computer. That’s a x40 speed improvement. Now, I don’t think twice before hitting ⌘R in TextMate. It feels liberating!"

(This was featured on the Rails Envy Podcast this past week which is where I found this.)

Paramedical

I installed the gem. However on starting the server... It blew up 'UNIXServer is required (LoadError)...' No support for windows ? – Shandeigh

Answer 2

When starting any tests, Rails first loads any fixtures you have (in test/fixtures) and recreates the database with them.

20-30 seconds sounds very slow though. Do you have a lot of fixtures that need to be loaded before your tests run, or is your database running slow?

Kolivas

Answer 3

Ruby's gem tool follows a path discovery algorithm which, apparently, is not Windows (as I see from your ruby -v) friendly.

You can get a clear picture if you trace, for example, a Rails application loading with ProcMon. Every (I really mean every) require starts a scan over all directories in Ruby's path plus all gem directories. A typical require takes 20 ms on an average machine. Since Rails makes hundreds of requires, those 20 ms easily sum up to seconds every time you launch the Rails environment. Take in the time to initialize the fixtures in the database and you get a better idea of why it takes so much time to just begin running the test-cases.

Perhaps because of each file-system architecture and implementation (path caching etc.), this is less of a problem in Linux than in Windows. I don't know who you should blame, though. It looks like the NTFS file-system could be improved with a better path caching implementation, but clearly the gem tool could implement the caching itself and have its performance not so dependent on the platform.

Belenbelesprit

Answer 4

It seems like Test::Unit is the simplest, but also one of the slowest ways to do unit testing with Ruby. One of alternatives is ZenTest.

Charlyncharm

Any docs on this one... also does it have any issues with Rails. e.g. Rails extends Test::Unit with some custom stuff. Everything works out ok? – Shandeigh

Answer 5

Test unit startup isn't particularly slow, and nowhere near 20 seconds.

(11:39) ~/tmp $ cat test_unit.rb 
require 'test/unit'
class MyTest < Test::Unit::TestCase
  def test_test
    assert_equal("this", "that")
  end
end

(11:39) ~/tmp $ time ruby test_unit.rb 
Loaded suite test_unit
Started
F
Finished in 0.007338 seconds.

  1) Failure:
test_test(MyTest) [test_unit.rb:4]:
<"this"> expected but was
<"that">.

1 tests, 1 assertions, 1 failures, 0 errors

real    0m0.041s
user    0m0.027s
sys     0m0.012s

It's probably something you're doing in your tests. Are you doing anything complicated? Setting up a database? Retrieving something from the internet?

Ego

Rails. its a Rails model test.. so yes it hits the database. But almost the entire time is before the Loaded messages.. after that the tests execute in like in a sec or 2 – Shandeigh

Answer 6

What does your test_helper.rb look like? Are you using instantiated fixtures?

self.use_instantiated_fixtures  = true

[edit]

If this is set to true try setting it to false.

Martyr

Nope. That's not it... tried that. Still 15+seconds to bootup – Shandeigh

Answer 7

Complete shot in the dark, but the majority of the time I see long startup times on things, it is usually due to some sort of reverse DNS lookup happening with some TCP socket communication somewhere along the way.

Try adding:

require 'socket'
Socket.do_not_reverse_lookup = true

at the top of your test file after your other require line.

Flinty

Crikey, I get down voted for taking a shot in the dark? If it didn't work, just say so. – Flinty

Thanks, someone did. Just about everytime I've had slow issues with Ruby, it was the Reverse DNS lookup thing, so I believe this was a well educated guess, and even labeled as so... I wonder why the downvote. Someone bring me back to equilibrium ;-) – Flinty