Skip tests on CRAN, but run locally

Question

It there an easy way to skip the execution of some tests in a package if the package is tested by CRAN? The background is, I like to have a lot of tests and in sum they are time consuming (not good for CRAN).

I know there is testthat::skip_on_cran() but I do not want to use package testthat to avoid another dependency. I am looking for an el-cheapo way to mimic testthat::skip_on_cran.

Ideally, I would like to have a testfile in directory pkg/tests that calls the tests (testfiles) and distuingishes if we are on cran or not:

if (!on_cran) {
 ## these tests are run only locally/when not on CRAN
 # run test 1 (e.g. source a file with a test)
 # run test 2
}
# run test 3 - always

Naivete

use env var defined locally, or split tests into multiple files and conditionally exit each with q(save = "no"). – Barcellona

Answer 1

Yes! You can handle this programmatically and automatically. Let me detail two ways I've set up:

Implicitly via version numbers: This is the approach taken by Rcpp for many years now, and it is entirely generic and not dependent on any other package. Our tests start from a file in tests/ and then hand over to RUnit, but that last part is an implementation detail.

In the main file tests/doRUnit.R we do this:

## force tests to be executed if in dev release which we define as
## having a sub-release, eg 0.9.15.5 is one whereas 0.9.16 is not
if (length(strsplit(packageDescription("Rcpp")$Version, "\\.")[[1]]) > 3) { 
    Sys.setenv("RunAllRcppTests"="yes")
}

In essence, we test if the version is of the form a.b.c.d -- and if so conclude that it is a development version. This implies "run all tests". Whereas a release version of the form a.b.c would go to CRAN, and not run these tests as they would exceed their time limit.

In each of the actual unit test files, we can then decide if we want to honour the variable and skip the test if set, or execute anyway:

.runThisTest <- Sys.getenv("RunAllRcppTests") == "yes"

if (.runThisTest) {

   ## code here that contains the tests

}

This mechanism is fully automatic, and does not depend on the user. (In the actual package version there is another if () test wrapped in there which allows us to suppress the tests, but that is a detail we do not need here).

I still like this approach a lot.

Explicitly via resource files Another package a few of us work on (a lot lately) requires a particular backend to be available. So in the Rblpapi package we tests for presence of a file which my coauthors and I each have below our $HOME directory in order to set up credentials and connection details. If the file is missing --- as e.g. on Travis CI, or CRAN, or for other users, the tests are skipped.

We chose to use the resource file as R file; it sources it if found and thereby sets values for options(). That way we can control directly whether to launch tests or not.

## We need to source an extra parameter file to support a Bloomberg connection
## For live sessions, we use ~/.Rprofile but that file is not read by R CMD check
## The file basically just calls options() and sets options as needed for blpHost,
## blpPort, blpAutoConnect (to ensure blpConnect() is called on package load) and,
## as tested for below, blpUnitTests.
connectionParameterFile <- "~/.R/rblpapiOptions.R"
if (file.exists(connectionParameterFile)) source(connectionParameterFile)

## if an option is set, we run tests. otherwise we don't.
## recall that we DO need a working Bloomberg connection...
if (getOption("blpUnitTests", FALSE)) {
  
    ## ... more stuff here which sets things up

}

Similarly to the first use case we can now set more variables which are later tested.

Explicitly via Travis CI Another option we use in rfoaas is to set the environment variable governing this in the Travis CI file:

env:
  global:
    - RunFOAASTests=yes

which the tests script then picks up:

## Use the Travis / GitHub integrations as we set this
## environment variable to "yes" in .travis.yml
##
## Set this variable manually if you want to run the tests
##
if (Sys.getenv("RunFOAASTests=yes") == "yes") runTests <- TRUE

In that case I also set the toggle based on my userid as I am pretty much the sole contributor to that project:

## Also run the tests when building on Dirk's box, even whem
## the environment variable is not set
if (isTRUE(unname(Sys.info()["user"])=="edd")) runTests <- TRUE

Explicitly via another variable You can of course also rely on another variable you use across all your packages. I find that to be a bad idea. If you set this in your shell, work on package A and set it to suppress tests but then switch to package B --- you will likely forget to unset the variable and then fail to test. I like this approach the least and do not use it.

Infrangible

Thanks! This is very comprehensive! Out of the approaches pointed out, I like the 'Implicitly via version numbers' best up until now. Simply because it does not require some "shared file/env var" among the package developers. Unfortunately, we have not defined a proper versioning scheme (yet)... I wonder if there is a direct way to query a well defined environment variable of CRAN machines - or if CRAN maintainers are willing to consider such a thing. – Naivete

In the past they always said no, and some folks on eg the mailing list (a particular R Core member whose initials are not BDR comes to mind) strongly oppose that. So I doubt it will happen, which is why I switched years ago to the implicit scheme, which just works which is what we want after all. – Infrangible

As a hint for those interested in the approach using the version number: numeric_version(packageDescription("pkg_name")$Version) is very handy if one wants to test for a specific version number, e.g. numeric_version(packageDescription("pkg_name")$Version)[[1,2]] gives the minor version as an integer and seems easier than doing the string splitting yourself (and handles "." and "-" as separators automatically). – Naivete

Answer 2

Use an environment variable, like testthat does:

skip_on_cran <- function() {
  if (identical(Sys.getenv("NOT_CRAN"), "true")) {
    return(invisible(TRUE))
  }

  skip("On CRAN")
}

Urgent

Thanks. Can you comment a bit more? So CRAN machines have an environment variable NOT_CRAN that is set to something other than "true"? Or shall one set one's own environment variable NOT_CRAN to "true" on the local machine? I would need that on different machines by various people, so if the latter is required, one would need some R code to check the machine... Do you know a package (on CRAN) that has this self-built check? – Naivete

@Naivete the NOT_CRAN environment variable will be absent on CRAN. If you make your tests conditional on that, they will not run. You could also specify some other random environment variable name; NOT_CRAN is just the variable used by testthat and devtools. You would need to set this on every machine you wanted the tests to run on. – Urgent

I see this a somewhat "indirect" approach. If there are multiple contributers to the package it becomes unhandy as everyone needs to have the proper setup for his local machine...isn't there some other way? Don't the CRAN machines somehow identify themselves, e.g. by an environment variable that we could use? Nothing is document in the CRAN policies but there is "Long-running tests and vignette code can be made optional for checking..." – Naivete

Nope. devtools sets the NOT_CRAN environment. If you don't want devtools/testthat you could distribute a Makefile with a test target that sets the relevant environment variable ... – Shotgun