Why do packages loaded inside `test_that()` provide their methods outside of `test_that()` and how can I prevent that?

Question

In my understanding, anything put inside test_that() should be compartmentalized, meaning that if I load a package in test_that(), its functions and methods shouldn't be available in other test_that() calls.

In the example below, there are 3 (empty) tests:

• In the first one, we can see that the method as.matrix.get_predicted is not available in the namespace.
• In the second one, I load the package insight, which provides the method as.matrix.get_predicted. In my understanding, this method should only be available in this test_that() call.
• In the third one, we can see that the method is available.

library(testthat)

test_that("foo 1", {
  print("as.matrix.get_predicted" %in% methods(as.matrix))
})
#> [1] FALSE
#> ── Skip (???): foo 1 ───────────────────────────────────────────────────────────
#> Reason: empty test

test_that("foo 2", {
  invisible(insight::get_parameters)
})
#> ── Skip (???): foo 2 ───────────────────────────────────────────────────────────
#> Reason: empty test

test_that("foo 3", {
  print("as.matrix.get_predicted" %in% methods(as.matrix))
})
#> [1] TRUE
#> ── Skip (???): foo 3 ───────────────────────────────────────────────────────────
#> Reason: empty test

Why is that? Are there some workarounds?

---

Edit: I'm looking for a solution specific to testthat, not another testing framework.

Rutan

In R, package loading is not scoped to functions. Running library() has global side effects, You'd have to explicitly unload the packages/namespaces yourself. Probably best dealt with via a test fixture: testthat.r-lib.org/articles/test-fixtures.html – Milan

Thanks, but it seems weird to me that there's no built-in functions to make sure of this. To me loading other packages in tests is quite standard but it clearly has side effects. I also saw that the package withr has with_package() and local_package() but they don't unload the package namespace so I don't really understand their purpose – Rutan

There is detach("package:parameters", unload = TRUE), but that won't unregister S3 methods or unload the dependencies of parameters that are loaded automatically by library(parameters). The correct approach is to run the relevant tests in a new R process, perhaps by having more than one *.R file under tests/ (what people who spurn testthat do). – Uralite

Even if I put the test that loads parameters in another test file (and if this test file is ran before the others), the S3 method will still be available in other test files so that doesn't work. – Rutan

Answer 1

Too long for a comment, but reproducible. The test files are sourced in collation order, each in a new R process, hence the library call in testB.R does not cause the test in testC.R to fail.

pkgname <- "testpackage"
testdir <- file.path(pkgname, "tests")

.add <- function(a, b) a + b
package.skeleton(pkgname, list = ".add")
dir.create(testdir, recursive = TRUE)
writeLines("stopifnot(!any(.S3methods(as.data.frame) == \"as.data.frame.lm\"))",
           file.path(testdir, "testA.R"))
writeLines("library(parameters); stopifnot(any(.S3methods(as.data.frame) == \"as.data.frame.lm\"))",
           file.path(testdir, "testB.R"))
writeLines("stopifnot(!any(.S3methods(as.data.frame) == \"as.data.frame.lm\"))",
           file.path(testdir, "testC.R"))

tools::Rcmd(c("check", pkgname))

* checking tests ...
  Running ‘testA.R’
  Running ‘testB.R’
  Running ‘testC.R’
 OK

This is the "vanilla" approach to compartmentalizing tests that many people still prefer over testthat and other frameworks, at least partly because it heeds the warning in ?detach:

The most reliable way to completely detach a package is to restart R.

and therefore is significantly easier for experts to debug and reason about, but that is perhaps a controversial opinion these days ...

Uralite

Thank you for your answer but I'm not going to replace testthat by another approach given the amount of tests. Surely I'm not the only one running into this issue so I guess there must be some testthat-specific solution – Rutan

It is a documented limitation of the R language: there is no reliable way to reverse the global effects of loading a package. If testthat had a magic answer, then it would be mentioned in vignette("test-fixtures"), but there simply isn't one, as stated in help("detach"). IMO, that vignette needs to be much more transparent about the limitations of testthat w.r.t. the global effects of loadNamespace calls. It is really a glaring omission. – Uralite

Answer 2

As a complement to @MikaelJagan's answer, there is a section in "R Packages" that addresses this problem:

It’s fair to say that library(somePkg) in the tests should be about as rare as taking a dependency via Depends, i.e. there is almost always a better alternative.

Unnecessary calls to library(somePkg) in test files have a real downside, because they actually change the R landscape. library() alters the search path. This means the circumstances under which you are testing may not necessarily reflect the circumstances under which your package will be used. This makes it easier to create subtle test bugs, which you will have to unravel in the future.

Rutan