How to cut a long ScalaTest spec to pieces
Asked Answered
F

4

1

I'm testing a REST API, and the code goes like this:

  1. Setting up stuff, populating a database using PUSH calls
  2. Testing API a
  3. Testing API b ...

The code is currently in one rather huge FlatSpec:

class RestAPITest extends FlatSpec
  with Matchers
  with ScalatestRouteTest
  with SprayJsonSupport

I would like to chop the "Testing API a/b/..." parts out, to have the code more manageable. Trying to do that seems like a no-no: what's the type of it - how to pass that on, etc. etc.

So, what's the recommended way to go about such stuff.

The a/b/... tests could be run in parallel, once the basic setup has succeeded.

I'm currently using assume within the a/b/... tests to make them cancel if the initialization failed.

Should I look at "fixtures" or what for this? Have tried BeforeAndAfterAll earlier, but didn't really get it working for me.

Thanks for the pointers / opinions. How do you keep your test suites short?

Ferrocyanide answered 3/12, 2014 at 13:21 Comment(2)
If the setup is always (mostly) the same, then I'd say using BeforeAndAfterAll is probably the way to go. What's not working for you regarding that?Fromenty
@KuluLimpa It's a while since I tried it. Will try again.Ferrocyanide
F
0

Adding as a new answer, so the differences are clear and the discussion above need not be removed. If I didn't do any typos, this should work (I did test it, and adopted in my project).

import org.scalatest._

/*
* Mix this trait into any specs that need 'TestA' to have been run first.
*/
trait TestAFirst {

  // Reading a 'TestA' object's field causes it to be instantiated and 'TestA' to be executed (but just once).
  //
  val testASuccess = TestA.success
}

/*
* 'TestA' gets instantiated via the companion object explicitly (thus @DoNotDiscover)
* and creates a success value field. Otherwise, it's a test just like any other.
*/
@DoNotDiscover
class TestA private extends FlatSpec {
  private var success = false   // read once, by the companion object

  behavior of "Root class"; {
    it should "run prior to any of the B,C classes" in {

      assert(true)    // ... A tests

      success = true
    }
  }
}

object TestA {
  val success = {
    val o= new TestA
    o.execute
    o.success   // getting a value from the executed test ('.execute()' itself doesn't provide a status)
  }
}

class TestB extends FlatSpec with TestAFirst {

  behavior of "class B"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... B tests
    }
  }
}

class TestC extends FlatSpec with TestAFirst {

  behavior of "class C"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... C tests
    }
  }
}
Ferrocyanide answered 7/12, 2014 at 20:7 Comment(0)
F
2

I'd say mixing in BeforeAndAfter or BeforeAndAfterAll are among the most intuitive ways to reduce duplication in a scenario where you want to do: "Setup" -> "run test1" -> "Setup" -> "run test2", "Setup" being (mostly) the same.

Suppose we have a nasty, hard to test Database:

object Database {
  private var content: List[Int] = Nil

  def add(value: Int) = content = value :: content

  def remove(): Unit = content = if (content.nonEmpty) content.tail else Nil

  def delete(): Unit = content = Nil

  def get: Option[Int] = content.headOption

  override def toString = content.toString()
}

It's a singleton (so we can't just instantiate a new Database for each test) and its mutable (so if the first test changes something, it would affect the second test).

Obviously, it would be more desireable to not have such a structure (e.g., it would be much nicer to work with the List that implements the Database in this example), but suppose we cannot simply change this structure.

Edit: Note that, in such a case, it is impossible (at least I can't think of a way) to run tests mutating the same singleton instance in parallel.

To still be able to test it, we need to have a clean state before running each test. Assuming we want to populate the database with the same values for each test, we could let our base testsuite class extend BeforeAndAfter. Note: There exists two traits: BeforeAndAfter, which defines before and after that are run before and after the execution of each test case, and BeforeAndAfterAll, which is different in that it defines methods that are run before and after each test suite.

class RestAPITest extends FlatSpec with ShouldMatchers with BeforeAndAfter {
  before {
    Database.delete()
    Database.add(4)
    Database.add(2)
  }
}

Now we can have a test suite ATest extend this base class:

class ATest extends RestAPITest {
  "The database" should "not be empty" in {
    Database.get shouldBe defined
  }
  it should "contain at least two entries" in {
    Database.remove()

    Database.get shouldBe defined
  }
  it should "contain at most two entries" in {
    Database.remove()
    Database.remove()

    Database.get should not be defined
  }
}

At the beginning of each test, the database contains the two values 4 and 2. We can now have other testsuits extend this base class:

class BTest extends RestAPITest {
  "The contents of the database" should "add up to 6" in {
    getAll.sum shouldBe 6
  }

  "After adding seven, the contents of the database" should "add up to 13" in {
    Database.add(7)

    getAll.sum shouldBe 13
  }

  def getAll: List[Int] = {
    var result: List[Int] = Nil
    var next = Database.get
    while(next.isDefined){
      result = next.get :: result
      Database.remove()
      next = Database.get
    }
    result
  }
}

Of course, we can also factor out common functionality in regular methods, as done in getAll which is used by both test cases.

Addendum:

Quote from the question:

How do you keep your test suites short?

Test code isn't much different from production code in my opinion. Factor out common functionality using methods and put them into separate traits if they don't belong to a specific class you already have.

However, if your production code requires the tests to execute always the same piece of code, then maybe there are too many dependencies in your production code. Say you have a function (in your production code)

def plus: Int = {
  val x = Database.get.get
  Database.remove()
  x + Database.get.get
}

then you cannot test this function unless you populate your database with the two values that you want to add. The best way to make your tests shorter and more readable in such a case would be to refactor your production code.

"plus 3 2" should "be 5" in {
  Database.add(3)
  Database.add(2)

  plus shouldBe 5
}

may become

"plus 3 2" should "be 5" in {
  plus(3,2) shouldBe 5
}

In some cases it's not easy to get rid of dependencies. But you may want your objects in a test scenario to depend on a special test environment. The database is a great example for that, as is the file system, or logging. Those things tend to be more costly in execution (I/O access) and may have themselves further dependencies that you must first establish.

In these cases, your tests will most likely profit from using mock objects. For example, you may like to implement an in-memory database that implements your database's interface.

Fromenty answered 3/12, 2014 at 15:44 Comment(2)
Your explanation of BeforeAndAfter etc. is fantastic - thank you. However, my need is different in the way that the database contents are purely read-only once the population step passes (which itself is a test). Mentioning deriving from RestAPITest made me try an approach where the test classes derive from a common base, and one keeps the others from executing via a normal latch mechanism. However :) since they are sisters (I first tried deriving from the population test but that kind of didn't work), they won't know which test should execute first, leading to a deadlock. Will try to fix it.Ferrocyanide
@Ferrocyanide Ah, in that case, you'd need code that is executed before all depending tests are executed - and executed only once. Maybe this question provides a solution: #15423837Fromenty
F
1

The way I got things to work is below.

I cause tests B and C to execute A before them, by mixing in a TestAFirst trait. That trait also makes sure TestA will only get executed once.

There are a couple of variations possible. I chose to disallow automatic launching of TestA itself by DoNotDiscover annotation. Ideally, I'd like to keep TestA look as much a normal test as possible, pushing all dependency handling into TestAFirst.

import java.util.concurrent.atomic.{AtomicBoolean}
import org.scalatest.{DoNotDiscover, FlatSpec}

/*
* Mix this trait into any specs that need 'TestA' to have been run first.
*/
trait TestAFirst extends FlatSpec {
  import TestAFirst._

  if (!doneTestA.getAndSet(true)) {
    // tbd. Can we detect here if 'execute' failed? Would be a better place to set 'testASuccess' than within the
    //      'TestA' itself (= limit all dependency things to 'TestAFirst').
    //
    (new TestA).execute
  }
}

object TestAFirst {
  val doneTestA= new AtomicBoolean
  @volatile var testASuccess= false   // remains 'false' if 'TestA' failed, causing B and C to cancel
}

/*
* 'TestA' is a test *almost* like any other.
*/
@DoNotDiscover
class TestA extends FlatSpec {
  import TestAFirst._

  behavior of "Root class"; {
    it should "run prior to any of the B,C classes" in {

      assert(true)    // ... A tests

      testASuccess = true
    }
  }
}

class TestB extends TestAFirst {
  import TestAFirst._

  behavior of "class B"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... B tests
    }
  }
}

class TestC extends TestAFirst {
  import TestAFirst._

  behavior of "class C"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... C tests
    }
  }
}

Better solutions are still welcome, but since this works I wanted to post it out. There are also other threads in SO (Doing something before or after all Scalatest tests and org.scalatest: Global setup (like beforeAllSuites?) ) that deal with similar issues, but not with a clear answer.

Naturally, the idea here is to place TestB, TestC etc. in different source files, getting to the modularity that I aimed for. This is just a snippet.

Ferrocyanide answered 5/12, 2014 at 12:55 Comment(2)
Thank you for adding your solution for future reference. While reading your solution, something similar came to my mind. What if you had a singleton object Setup that executes all your setup code as part of its instantiation? You could then create a trait that simply accesses this singleton object and mix it in with your tests. The singleton property would then guarantee that the setup is executed exactly once and mixing in the object access should guarantee that it is executed before the tests are run. Then, you wouldn't have to make TestA special.Fromenty
That's awesome advice, @KuluLimpa and I think I will try exactly that. It's using a bit of Scala "magic" (s.a. relying on lazy object initialization) but it does make sense. Those things are well defined so I'll just add comments to others to realize what's going on. Will mention here tomorrow how it went.Ferrocyanide
F
0

Adding as a new answer, so the differences are clear and the discussion above need not be removed. If I didn't do any typos, this should work (I did test it, and adopted in my project).

import org.scalatest._

/*
* Mix this trait into any specs that need 'TestA' to have been run first.
*/
trait TestAFirst {

  // Reading a 'TestA' object's field causes it to be instantiated and 'TestA' to be executed (but just once).
  //
  val testASuccess = TestA.success
}

/*
* 'TestA' gets instantiated via the companion object explicitly (thus @DoNotDiscover)
* and creates a success value field. Otherwise, it's a test just like any other.
*/
@DoNotDiscover
class TestA private extends FlatSpec {
  private var success = false   // read once, by the companion object

  behavior of "Root class"; {
    it should "run prior to any of the B,C classes" in {

      assert(true)    // ... A tests

      success = true
    }
  }
}

object TestA {
  val success = {
    val o= new TestA
    o.execute
    o.success   // getting a value from the executed test ('.execute()' itself doesn't provide a status)
  }
}

class TestB extends FlatSpec with TestAFirst {

  behavior of "class B"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... B tests
    }
  }
}

class TestC extends FlatSpec with TestAFirst {

  behavior of "class C"; {
    it should "run after A has been run" in {
      assume(testASuccess)

      assert(true)    // ... C tests
    }
  }
}
Ferrocyanide answered 7/12, 2014 at 20:7 Comment(0)
S
-1

You use Spray framework? You can try spray.testkit.Specs2RouteTest

   class RestAPISpec extends Specification with Specs2RouteTest {
     "RestAPITest" should {
        "Test A" in {
          ... some code
        }
        "Test B" in {
          ... some code
        }
     }
   }
Suburbanize answered 3/12, 2014 at 14:13 Comment(1)
ScalatestRouteTest does the same, using ScalaTest (which is more familiar to me). Would something with regard to the question be better with Spec2? Your sample does not show snipping some of the "some code" away from the main body - that's what I am trying to do.Ferrocyanide

© 2022 - 2024 — McMap. All rights reserved.