Create assembly jar that contains all tests in SBT project+subprojects

Question

I have an interesting problem where I basically need to create a .jar (plus all of the classpath dependencies) that contains all of the tests of an SBT project (plus any of its subprojects). The idea is that I can just run the jar using java -jar and all of the tests will execute.

I heard that this is possible to do with sbt-assembly but you would have to manually run assembly for each sbt sub-project that you have (each with their own .jars) where as ideally I would just want to run one command that generates a giant .jar for every test in every sbt root+sub project that you happen to have (in the same way if you run test in an sbt project with sub projects it will run tests for everything).

The current testing framework that we are using is specs2 although I am not sure if this makes a difference.

Does anyone know if this is possible?

Weiler

Answer 1

Exporting test runner is not supported

sbt 1.3.x does not have this feature. Defined tests are executed in tandem with the runner provided by test frameworks (like Specs2) and sbt's build that also reflectively discovers your defined tests (e.g. which class extends Spec2's test traits?). In theory, we already have a good chunk of what you'd need because Test / fork := true creates a program called ForkMain and runs your tests in another JVM. What's missing from that is dispatching of your defined tests.

Using specs2.run runner

Thankfully Specs2 provides a runner out of the box called specs2.run (See In the shell):

scala -cp ... specs2.run com.company.SpecName [argument1 argument2 ...]

So basically all you need to know is:

1. your classpath
2. list of fully qualified name for your defined tests

Here's how to get them using sbt:

> print Test/fullClasspath
* Attributed(/private/tmp/specs-runner/target/scala-2.13/test-classes)
* Attributed(/private/tmp/specs-runner/target/scala-2.13/classes)
* Attributed(/Users/eed3si9n/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.13/1.2.0/scala-xml_2.13-1.2.0.jar)
...
> print Test/definedTests
* Test foo.HelloWorldSpec : subclass(false, org.specs2.specification.core.SpecificationStructure)

We can exercise specs2.run runner from sbt shell as follows:

> Test/runMain specs2.run foo.HelloWorldSpec

Aggregating across subprojects

Aggregating tests across subprojects requires some thinking. Instead of creating a giant ball of assembly, I would recommend the following. Create a dummy subproject testAgg, and then collect all the Test/externalDependencyClasspath and Test/packageBin into its target/dist. You can then grab all the JAR and run java -jar ... as you wanted.

How would one go about that programmatically? See Getting values from multiple scopes.

lazy val collectJars = taskKey[Seq[File]]("")
lazy val collectDefinedTests = taskKey[Seq[String]]("")
lazy val testFilter = ScopeFilter(inAnyProject, inConfigurations(Test))

lazy val testAgg = (project in file("testAgg"))
  .settings(
    name := "testAgg",
    publish / skip := true,
    collectJars := {
      val cps = externalDependencyClasspath.all(testFilter).value.flatten.distinct
      val pkgs = packageBin.all(testFilter).value
      cps.map(_.data) ++ pkgs
    },
    collectDefinedTests := {
      val dts = definedTests.all(testFilter).value.flatten
      dts.map(_.name)
    },
    Test / test := {
      val jars = collectJars.value
      val tests = collectDefinedTests.value
      sys.process.Process(s"""java -cp ${jars.mkString(":")} specs2.run ${tests.mkString(" ")}""").!
    }
  )

This runs like this:

> testAgg/test
[info] HelloWorldSpec
[info]
[info] The 'Hello world' string should
[info]   + contain 11 characters
[info]   + start with 'Hello'
[info]   + end with 'world'
[info]
[info]
[info] Total for specification HelloWorldSpec
[info] Finished in 124 ms
3 examples, 0 failure, 0 error
[info] testAgg / Test / test 1s

If you really want to you probably could generate source from the collectDefinedTests make testAgg depend on the Test configurations of all subprojects, and try to make a giant ball of assembly, but I'll leave as an exercise to the reader :)

Sparry

Thanks for the informative answer, will definitely look into this. Basically our use case is that we don't want to run SBT in order for us to run all of our tests, do you think it makes sense to official support this in some better way? I will also try to make an SBT plugin to automate this in some way. – Weiler

Yea. I'd say this would be an interesting plugin idea, especially if you can make it work for all test frameworks, which hopefully isn't too difficult since testOnly exists. – Sparry