Is it possible to determine test order in testthat?

Question

I am using testthat to check the code in my package. Some of my tests are for basic functionality, such as constructors and getters. Others are for complex functionality that builds on top of the basic functionality. If the basic tests fail, then it is expected that the complex tests will fail, so there is no point testing further.

Is it possible to:

1. Ensure that the basic tests are always done first
2. Make a test-failure halt the testing process

Stroh

Answer 1

To answer your question, I don't think it can be determined other than by having appropriate alphanumeric naming of your test-*.R files.

From testthat source, this is the function which test_package calls, via test_dir, to get the tests:

find_test_scripts <- function(path, filter = NULL, invert = FALSE, ...) {
  files <- dir(path, "^test.*\\.[rR]$", full.names = TRUE)

What is wrong with just letting the complex task fail first, anyway?

Seif

"What is wrong ..."? My first thought is "time to compute", looking for a fast-fail vice long-wait. (One of my projects unavoidably requires 15-20 minutes to run through all the tests. Additionally I have dependencies between tests: one test sets up the environment, other tests reuse components. Breaking tests into multiple files is useful in my case, admittedly not all.) – Melisa

Fair point. It is tricky but possible to divide tests, only some of which run on CRAN testing, if that is important to you. Running a particular test suite requires just a command line argument, but doesn't integrate at all with Rstudio, and sharing test content between two groups, or using testthat, would probably all just require tons of maintenance and break things. Thus I agree with you! – Seif

Answer 2

Regarding tests order, you can use testthat config in the DESCRIPTION file to determine test order (by file) - As the documentation suggests

By default testthat starts the test files in alphabetical order. If you have a few number of test files that take longer than the rest, then this might not be the best order. Ideally the slow files would start first, as the whole test suite will take at least as much time as its slowest test file. You can change the order with the Config/testthat/start-first option in DESCRIPTION. For example testthat currently has:

Config/testthat/start-first: watcher, parallel*

Docs

Disappear

Answer 3

A recent(ish) development to testthat is parallel test processing

This might not be suitable in your case as it sounds like you might have complex interdependencies between tests. If you could isolate your tests (I think that would be the more usual case anyway) then parallel processing is a great solution as it should speed up overall processing time as well as probably showing you the 'quick' tests failing before the 'slow' tests have completed.

Wellesley