Spring vs. Zeus performance issue: Spring not running tests much faster after 1st run?

Question

So I'm trying to improve my test execution time, and have been following some of the excellent advice out there (most notably, Railscasts 412).

Zeus seems to be working fine (other than the weirdness that requires you to type zeus test test in order to run the Minitest suite.)

Here's the timed output from zeus:

Baseline (rake test):

$ time rake test
Run options: --seed 62848

# Running tests:

........

Finished tests in 0.117071s, 68.3346 tests/s, 68.3346 assertions/s.

8 tests, 8 assertions, 0 failures, 0 errors, 0 skips

real    0m5.771s
user    0m4.477s
sys     0m0.872s

And with Zeus:

$ time zeus test test
Run options: --seed 10325

# Running tests:

........

Finished tests in 0.126365s, 63.3087 tests/s, 63.3087 assertions/s.

8 tests, 8 assertions, 0 failures, 0 errors, 0 skips

real    0m0.765s
user    0m0.209s
sys     0m0.030s

Good. 5.7s down to 0.7s.

Now, with spring, I'm just not seeing the improvement after the first run. I'm running Spring gem 1.1.0beta4, installing as-described (adding to Gemfile, then running bundle exec spring binstub --all).

And here's what I've got:

Baseline, 1st execution (rake test):

time bin/rake test
Run options: --seed 49915

# Running tests:

........

Finished tests in 0.108359s, 73.8287 tests/s, 73.8287 assertions/s.

8 tests, 8 assertions, 0 failures, 0 errors, 0 skips

real    0m6.265s
user    0m0.072s
sys     0m0.020s

Confirm that spring is running:

$ spring status
Spring is running:

 2990 spring server | gdc4 | started 43 secs ago  
 2991 spring app    | gdc4 | started 43 secs ago | test mode    

And run the tests again...

$ time bin/rake test
Run options: --seed 27416

# Running tests:

........

Finished tests in 0.132677s, 60.2968 tests/s, 60.2968 assertions/s.

8 tests, 8 assertions, 0 failures, 0 errors, 0 skips

real    0m3.885s
user    0m0.069s
sys     0m0.014s

So a bit of a speedup, but still almost 4 seconds to execute. Meanwhile, btw, the rails console DOES load up a ton faster w/ Spring....< 1 second.

So, my question is: Why is Zeus able to get the order-of-magnitude speedup on tests (5.7s -> 0.7s) but Spring is not?

Running Rails 4.0.2, Ruby 2.0.0 btw.

Groundsill

Answer 1

You can't run

rake test

you need to run

spring rake test

Example:

time rake -T

10.737s

time rake -T

10.811s # no improvement, as you noted

time spring rake -T

18.468s

time spring rake -T

1.140s # huge improvement

Kwon

Answer 2

I've only played with spring a little bit and haven't compared times against zeus (which I prefer) but my guess is you're still seeing the slow [ruby] startup time of spring itself. Zeus doesn't have any of that since it's written in Go.

Divergent

Answer 3

I see the same difference here. Running the rails server for instance: with zeus < 1s with spring ~ 4s

Magness

Answer 4

Probably because zeus test doesn't run db:reset.

On one of my applications, Spring runs tests in 5-6s, whereas Zeus runs them in about 0.5s, and it's definitely due to Spring reloading the database each time.

Olympic