How can a test script inform R CMD check that it should emit a custom message?

Question

I'm writing a R package (delftfews) here at office. we are using svUnit for unit testing.

our process for describing new functionality: we define new unit tests, initially marked as DEACTIVATED; one block of tests at a time we activate them and implement the function described by the tests. almost all the time we have a small amount of DEACTIVATED tests, relative to functions that might be dropped or will be implemented.

my problem/question is: can I alter the doSvUnit.R so that R CMD check pkg emits a NOTE (i.e. a custom message "NOTE" instead of "OK") in case there are DEACTIVATED tests?

as of now, we see only that the active tests don't give error:

.
.
* checking for unstated dependencies in tests ... OK
* checking tests ...
  Running ‘doSvUnit.R’
 OK
* checking PDF version of manual ... OK

which is all right if all tests succeed, but less all right if there are skipped tests and definitely wrong if there are failing tests. In this case, I'd actually like to see a NOTE or a WARNING like the following:

.
.
* checking for unstated dependencies in tests ... OK
* checking tests ...
  Running ‘doSvUnit.R’
 NOTE
6 test(s) were skipped.
 WARNING
1 test(s) are failing.
* checking PDF version of manual ... OK

As of now, we have to open the doSvUnit.Rout to check the real test results.

---

I contacted two of the maintainers at r-forge and CRAN and they pointed me to the sources of R, in particular the testing.R script.

if I understand it correctly, to answer this question we need patching the tools package:

• scripts in the tests directory are called using a system call,
• output (stdout and stderr) go to one single file,
• there are two possible outcomes: ok or not ok,

so I opened a change request on R, proposing something like bit-coding the return status, bit-0 for ERROR (as it is now), bit-1 for WARNING, bit-2 for NOTE.

with my modification, it would be easy producing this output:

.
.
* checking for unstated dependencies in tests ... OK
* checking tests ...
  Running ‘doSvUnit.R’
 NOTE - please check doSvUnit.Rout.
 WARNING - please check doSvUnit.Rout.
* checking PDF version of manual ... OK

---

Brian Ripley replied "There are however several packages with properly written unit tests that do signal as required. Please do take this discussion elsewhere: R-bugs is not the place to ask questions." and closed the change request.

---

anybody has hints?

Misti

What do you mean by NOTE? As in a sound from the computer speakers? – Beneficence

edited question to explain NOTE, opposed to OK but not so bad as WARNING. – Misti

just a note: I've implemented a solution to a somewhat opposite problem, that is, being aware in unit testing of the examples in the manual pages. it is part of svUnit. – Misti

Brian Ripley was probably referring to the R-devel mailing list. – Mcbee

makes sense, I will probably do that. it would be nice if some of the 20 upvoters would support the change request (or give more feedback here). – Misti

since my initial issue was related to making failing and skipped unit tests visible, I'm putting this short sed script here: sed ./pkg.Rcheck/tests/report.xml -e '/<testcase.*\/>/d' -e '/testcase.*time="NA"/,/\/testcase/d' -e '/testcase/s/"\([A-Za-z][a-zA-Z0-9._]*\)"/"\x1b[1;31m\1\x1b[0m"/g' -e '{s/&quot;/"/g;s/&lt;/</g;s/&gt;/>/g;s/&apos;/\x27/g;}'. it helps make failures more visible. – Misti

Answer 1

You should be able to alter the doSvUnit.R script to at least emit warnings and errors as you describe. What you want to do is to run the tests and then inspect the return value of the test runner and have R code that calls, warning() or stop().

For an example of how this was done using RUnit, take a look at the codetoolsBioC package in the Bioconductor repository. The relevant code is in inst/templates and copied below:

.test <- function(dir, pattern = ".*_test\\.R$")
{
    .failure_details <- function(result) {
        res <- result[[1L]]
        if (res$nFail > 0 || res$nErr > 0) {
            Filter(function(x) length(x) > 0,
                   lapply(res$sourceFileResults,
                          function(fileRes) {
                              names(Filter(function(x) x$kind != "success",
                                           fileRes))
                          }))
        } else list()
    }

    if (missing(dir)) {
        dir <- system.file("unitTests", package="@PKG@")
        if (!nzchar(dir)) {
            dir <- system.file("UnitTests", package="@PKG@")
            if (!nzchar(dir))
                stop("unable to find unit tests, no 'unitTests' dir")
        }
    }

    ## Run unit tests from the directory containing the test files.
    ## This allows tests to refer to data files with relative paths
    cwd <- getwd()
    on.exit(setwd(cwd))
    setwd(dir)

    require("RUnit", quietly=TRUE) || stop("RUnit package not found")
    RUnit_opts <- getOption("RUnit", list())
    RUnit_opts$verbose <- 0L
    RUnit_opts$silent <- TRUE
    RUnit_opts$verbose_fail_msg <- TRUE
    options(RUnit = RUnit_opts)
    suite <- defineTestSuite(name="@PKG@ RUnit Tests", dirs=getwd(),
                             testFileRegexp=pattern,
                             rngKind="default",
                             rngNormalKind="default")
    result <- runTestSuite(suite)
    cat("\n\n")
    printTextProtocol(result, showDetails=FALSE)
    if (length(details <- .failure_details(result)) >0) {
        cat("\nTest files with failing tests\n")
        for (i in seq_along(details)) {
            cat("\n  ", basename(names(details)[[i]]), "\n")
            for (j in seq_along(details[[i]])) {
                cat("    ", details[[i]][[j]], "\n")
            }
        }
        cat("\n\n")
        stop("unit tests failed for package @PKG@")
    }
    result
}

Pantaloon

well, I have added to svUnit the code that allows it to produce a complete xml report of the test cases, but I don't think you can provide the feedback I describe without altering check.R. if you say I should be able to do so, grab the delftfews project, it currently contains a couple of <skipped> and <failure> tests and test your idea, I don't get it from the example you provide. but thanks for the effort. – Misti

seth, did you test what you suggest? according to me, the output of this just goes into the <filename>out log, not to console. – Misti

Answer 2

I contacted two of the maintainers at r-forge and CRAN and they pointed me to the sources of R, in particular the check.R script.

if I understand it correctly:

• scripts in the tests directory are called using a system call,
• output (stdout and stderr) go to one single file,
• there are two possible outcomes: ok or not ok,
• to answer this question we need patching the library/tools package.

---

I opened a change request on R, my first guess is something like bit-coding the return status, bit-0 for ERROR (as it is now), bit-1 for WARNING, bit-2 for NOTE.

from doSvUnit.R, I would quit with status 2 in case of failures and 4 in case of skipped tests.

the patch would look like this:

Index: src/library/tools/R/testing.R
===================================================================
--- src/library/tools/R/testing.R   (revision 57214)
+++ src/library/tools/R/testing.R   (working copy)
@@ -352,10 +352,16 @@
         } else
             cmd <- paste("LANGUAGE=C", "R_TESTS=startup.Rs", cmd)
         res <- system(cmd)
-        if (res) {
+        if (res%/%4 %% 2) {
+            message("NOTE")
+        }
+        if (res%/%2 %% 2) {
+            message("WARNING")
+        }
+        if (res %% 2) {
             file.rename(outfile, paste(outfile, "fail", sep="."))
             return(1L)
         }
         savefile <- paste(outfile, "save", sep = "." )
         if (file.exists(savefile)) {
             message("  Comparing ", sQuote(outfile), " to ",

unpatched R sees anything different from 0 as ERROR.

Misti