ScalaTest in sbt: is there a way to run a single test without tags?

Question

I know that a single test can be ran by running, in sbt,

testOnly *class -- -n Tag

Is there a way of telling sbt/scalatest to run a single test without tags? For example:

testOnly *class -- -X 2

it would mean "run the second test in the class. Whatever it is". We have a bunch of tests and no one bothered to tag them, so is there a way to run a single test without it having a tag?

Precedence

Your subject says "Scalatest-sbt". When sbt is hyphenated people would normally think of a plugin. But, just to clarify, you're talking about using ScalaTest from a modern version of sbt like sbt 0.12, not joshcough/scalatest-sbt a plugin for sbt 0.5.2-p3 written 4 years ago. Correct? – Thaumaturgy

Correct. This is an old question and I have since then figured out that no, it is not possible (as far as I know). I haven't closed it in case someone did in fact managed to figure out a way, but I no longer need this to be answered. – Precedence

There is a thread on this (with participation from both Bill Venners and Mark Harrah) at groups.google.com/forum/#!topic/scalatest-users/1oRMkudFAXM , but no solution yet – Lachrymatory

There's also a case for general support on running a single test sbt#911 (github.com/sbt/sbt/issues/911). – Thaumaturgy

Note that if you're running from the command line, you have to enclose everything after sbt in quotes, e.g. sbt "test-only *SingleTestSuite" – Alberic

the quotes are shell syntax, preventing sbt from interpreting the single command as a series of commands, while also preventing the shell from expanding *. MySuite is a class name – Lachrymatory

Answer 1

This is now supported (since ScalaTest 2.1.3) within interactive mode:

testOnly *MySuite -- -z foo

to run only the tests whose name includes the substring "foo".

For exact match rather than substring, use -t instead of -z.

---

If you run it from the command line, it should be as single argument to sbt:

sbt 'testOnly *MySuite -- -z foo'

Lachrymatory

@SethTisue Would you be able to post a working example which uses -t for exact matching? I am unable to get it to work. – Storax

@Storax gist.github.com/SethTisue/f75cd8b72128ba0a0a81. (if this helps you fix your problem, let me know how I should update my answer.) – Lachrymatory

@SethTisue Thanks for the link. I tried this and it doesn't work on my particular project even though -z does. Not a big deal. Mine is a play app using org.scalatestplus version 1.1.0 so maybe it's related to that. – Storax

For JUnitSuite in Scala, it doesn't work. Do you know what kind of options should I use? – Turrell

In case anyone wants to run a specific integration test (supposedly placed under src/it), they need to prepend it to testOnly. For instance, on the command line: sbt "it:testOnly *MyIntegrationTestSuite". – Wetmore

How can I filter on multiple substrings? Tests may be grouped in a hierarchy (WordSpec), and the name parts separated by when and should can repeat between tests. To chose one specific test I need to say "name contains this AND that". – Countdown

can you further explain this syntax? what is the quote near testOnly? What is MySuite? should that be the name of the project? – Evince

the single quotes have two purposes: to tell sbt that the whole thing is a single command and not a series of them, and to prevent the shell from interpreting the * as a glob. MySuite is the name of a test class – Lachrymatory

Answer 2

I wanted to add a concrete example to accompany the other answers

You need to specify the name of the class that you want to test, so if you have the following project (this is a Play project):

You can test just the Login tests by running the following command from the SBT console:

test:testOnly *LoginServiceSpec

If you are running the command from outside the SBT console, you would do the following:

sbt "test:testOnly *LoginServiceSpec"

Hagiology

Upvote because apparently the double quotes are necessary: sbt "test:testOnly *LoginServiceSpec" – Emolument

Most useful answer for me here. 👍 But the commands can be slightly simplified; in SBT console: testOnly *LoginServiceSpec, and outside: sbt "testOnly *LoginServiceSpec" – Pylorus

It seems that this is a sbt 0.13 syntax. Either use sbt "Test/testOnly *LoginServiceSpec or event better as mentioned elsewhere sbt "testOnly *LoginServiceSpec" – Rebuff

Answer 3

I don't see a way to run a single untagged test within a test class but I am providing my workflow since it seems to be useful for anyone who runs into this question.

From within a sbt session:

test:testOnly *YourTestClass

(The asterisk is a wildcard, you could specify the full path com.example.specs.YourTestClass.)

All tests within that test class will be executed. Presumably you're most concerned with failing tests, so correct any failing implementations and then run:

test:testQuick

... which will only execute tests that failed. (Repeating the most recently executed test:testOnly command will be the same as test:testQuick in this case, but if you break up your test methods into appropriate test classes you can use a wildcard to make test:testQuick a more efficient way to re-run failing tests.)

Note that the nomenclature for test in ScalaTest is a test class, not a specific test method, so all untagged methods are executed.

If you have too many test methods in a test class break them up into separate classes or tag them appropriately. (This could be a signal that the class under test is in violation of single responsibility principle and could use a refactoring.)

Operand

for who are facing "No tests were executed": *YourTestClass must be class name. Not file name. – Nitrobacteria

it was testOnly instead of test-only for me. – Pirouette

Like the explanation of what the * meant. – Poleyn

Answer 4

Just to simplify the example of Tyler.

test:-prefix is not needed.

So according to his example:

In the sbt-console:

testOnly *LoginServiceSpec

And in the terminal:

sbt "testOnly *LoginServiceSpec"

Hagiographer

Answer 5

Here's the Scalatest page on using the runner and the extended discussion on the -t and -z options.

This post shows what commands work for a test file that uses FunSpec.

Here's the test file:

package com.github.mrpowers.scalatest.example

import org.scalatest.FunSpec

class CardiBSpec extends FunSpec {

  describe("realName") {

    it("returns her birth name") {
      assert(CardiB.realName() === "Belcalis Almanzar")
    }

  }

  describe("iLike") {

    it("works with a single argument") {
      assert(CardiB.iLike("dollars") === "I like dollars")
    }

    it("works with multiple arguments") {
      assert(CardiB.iLike("dollars", "diamonds") === "I like dollars, diamonds")
    }

    it("throws an error if an integer argument is supplied") {
      assertThrows[java.lang.IllegalArgumentException]{
        CardiB.iLike()
      }
    }

    it("does not compile with integer arguments") {
      assertDoesNotCompile("""CardiB.iLike(1, 2, 3)""")
    }

  }

}

This command runs the four tests in the iLike describe block (from the SBT command line):

testOnly *CardiBSpec -- -z iLike

You can also use quotation marks, so this will also work:

testOnly *CardiBSpec -- -z "iLike"

This will run a single test:

testOnly *CardiBSpec -- -z "works with multiple arguments"

This will run the two tests that start with "works with":

testOnly *CardiBSpec -- -z "works with"

I can't get the -t option to run any tests in the CardiBSpec file. This command doesn't run any tests:

testOnly *CardiBSpec -- -t "works with multiple arguments"

Looks like the -t option works when tests aren't nested in describe blocks. Let's take a look at another test file:

class CalculatorSpec extends FunSpec {
  it("adds two numbers") {
    assert(Calculator.addNumbers(3, 4) === 7)
  }
}

-t can be used to run the single test:

testOnly *CalculatorSpec -- -t "adds two numbers"

-z can also be used to run the single test:

testOnly *CalculatorSpec -- -z "adds two numbers"

See this repo if you'd like to run these examples. You can find more info on running tests here.

Vinegar

Wow man! This is really an innovative test suite I have encountered in my life. Hats off. – Keller