Testing multiple hosts with the same test using serverspec

Question

The Advanced Tips section of the Serverspec site shows an example of testing multiple hosts with the same test set. I've built an example of my own (https://gist.github.com/neilhwatson/81249ad393800a76a8ad), but there are problems.

The first problem is that the tests stop at the first failure rather than proceeding through the lot and keeping a tally. The second is that the failure output does not indicate on which host the failure occurred. What can I do to fix these problems and produce a final report for all hosts?

Recension

Answer 1

For the first issue, ServerSpec by default will run all your tests. However, since you have a loop that executes a Rake task for each environment, the first environment to have a failure causes the task to fails and so an exception is raised and the rest of your tasks don't run.

I've forked your gist and updated the Rake task to surround it with a begin/rescue.

...
begin
  desc "Run serverspec to #{host}"
  RSpec::Core::RakeTask.new(host) do |t|
    ENV['TARGET_HOST'] = host
    t.pattern = "spec/base,cfengine3/*_spec.rb"
  end
rescue
end
...

For the second problem, it doesn't look like ServerSpec will output which environment the tests are running in. But since the updated Gist shows that the host gets set in the spec_helper.rb we can use that to add an RSpec configuration that sets up an after(:each) and only output the host on errors. The relevant code changes are in a fork of the gist, but basically you'll just need the below snippet in your spec_helper.rb:

RSpec.configure do |c|
  c.after(:each) do |example|
    if example.exception
       puts "Failed on #{host_run_on}"
    end
  end
end

Retrace

Closer, but I still can't tell what host failure happens on. New gist: gist.github.com/neilhwatson/1d41c696102c01bbb87a – Recension

A new error: example is not available from within an example (e.g. an it block) or from constructs that run in the scope of an example (e.g. before, let, etc). It is only available on an example group (e.g. a describe or context block). – Recension

Ahh, my bad, that's from RSpec 2, not RSpec 3. It's done this way in RSpec 3: https://mcmap.net/q/548316/-detect-rspec-test-failure-on-after-each-method. I'll update the response. – Retrace

Nope: gist.github.com/neilhwatson/… – Recension

OK, that tells me host is nil. Can you try the way I created a host_run_on temporary variable instead? gist.github.com/amaltson/… – Retrace

Still returns nil hostname. – Recension

So I don't get it... how does ServerSpec get that host name. If the value is nil, how does ServerSpec know which host to log into? – Retrace