testthat pattern for long-running tests

Question

I have a bunch of tests that I don't want them running during CRAN checks or Travis CI builds. They are either long-running, or they could cause transaction/concurrency conflicts writing to a networked database. What approach of separating them (from the R CMD check tests) works best with testthat?

Should I put those tests in a separate folder? Should I tag their filename and use a regex? (eg Using filter argument in test_package to skip tests by @Jeroen)

http://cran.r-project.org/web/packages/policies.html:

Long-running tests and vignette code can be made optional for checking, but do ensure that the checks that are left do exercise all the features of the package.

Soule

If you put them in another directory within tests, then you can still test them manually with test_dir(), but they won't be running with test() or R CMD check. E.g. R6 has some manual tests: github.com/wch/R6/tree/master/tests – Galinagalindo

I like that @GaborCsardi, and how the tests are contained together, but distinguished by test() and test_dir(). – Soule

@GaborCsardi that was exactly what I was looking for, and it worked well for my package's needs. If you change your comment to an answer, I'd like to give your response credit. – Soule

Answer 1

If you put them in another directory within tests, then you can still test them manually with test_dir(), but they won't be running with test() or R CMD check.

E.g. R6 has some manual tests: https://github.com/wch/R6/tree/master/tests

Galinagalindo

Answer 2

FYI: testthat 0.9 now comes with a skip() function.

However, I can't seem to figure out how/where exactly to use it. Putting inside my test_that() function, the test runs anyway. If you put it before testthat(), then skip() throws an error.

Mchail

Thanks for mentioning that --I saw the release announcement a few days ago. Immediately inside the test functions, I've added the like testthat::skip_on_cran(). It's a good fit for some of my tests, because the main reason I wanted some manual tests are so that it wouldn't get rejected from CRAN if the server wasn't peppy right then. I don't really care if the Travis-CI or R-Forge checks fail, because it's easy to rerun. – Soule

Could you tell me, how exactly you are using skip() and/or skip_on_cran()? Where do I need to put it in order to make a specific test skip when running test_package() or CRTL+SHFT+T in RStudio? – Mchail

Here is a link to the test code. To run it, I either check the package with CTRL+SHIFT+E or run just the tests with test_results_checked <- devtools::test(). – Soule

Thanks man. That's how I did it and it didn't work for some reason but now it does ;-) Must have been something else – Mchail

Note that skip (At least in current dev version) requires a message argument. E.g. inside test_that( use if (is.friday(today())) skip("Not today.") – Dwight