testthat fails within devtools::check but works in devtools::test

Question

Is there any way to reproduce the environment which is used by devtools::check?

I have the problem that my tests work with devtools::test() but fail within devtools::check(). My problem is now, how to find the problem. The report of check just prints the last few lines of the error log and I can't find the complete report for the testing.

checking tests ... ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
...

I know that check uses a different environment compared to test but I don't know how I should debug these problems since they are not reproducible at all. Specially these test where running a few month ago, so not sure where to look for the problem.

EDIT

actually I tried to locate my problem and I found a solution. But to post my solution to it, I have to add more details.

So my test always failed since I was testing a markdown script if it is running without errors and afterwards I was checking if some of the environmental variables are set correctly. These where results which I calculate with the script as well as standard settings which I set. So I wanted to get a warning if I forgot to change some of my settings after developing...

Anyway, since it is a markdown script, I had to extract the code and I was using comments from this post knitr: run all chunks in an Rmarkdown document using knitr::purl to get the code and sys.source to execute it.

runAllChunks <- function(rmd, envir=globalenv()){
  # as found here https://stackoverflow.com/questions/24753969
  tempR <- tempfile(tmpdir = '.', fileext = ".R")
  on.exit(unlink(tempR))
  knitr::purl(rmd, output=tempR, quiet=TRUE)
  sys.source(tempR, envir=envir)
}

For some reason, this produces an error since maybe a few weeks (not sure which new packages I installed lately...). But since there is a new comment, that I can just use knitr::knit which also executes the code, this worked as expected and now my test no longer complains.

So in the end, I don't know where the problem exactly was, but this is now working.

Therrien

Answer 1

I recently had a similar issue with my tests breaking (succeeding with devtools::test() but failing with devtools::check()). I don't know if this solution necessarily fixes the problem above, but it should help to track down similar problems.

In my case, the problem ultimately came down to using a function that needed a package listed in Suggests rather than in Imports/Depends. In particular, my function called httr::content(), which broke when I tried to pass it the as = "parsed" argument. It turns out that as = "parsed" uses a suggested package, readr to read a csv, and I needed to add it to my dependencies for devtools::check() to work.

Priority

Could you elaborate on that, doesn't this merit PR on devtools? I'm facing a similar problem and it's not easy issue to fix. – Whitver

I didn't think of it at the time (I only thought to record my experience here), but it probably is worth at least opening an issue about throwing better error messages. To clarify: what I saw was that problems arise if you depend on package A, but use a function from A that directly uses package B. If A suggests B, then it will pass the check. But if B is not declared in your dependencies, test() will work because it's interactive but check() will fail because it starts from a totally clean slate. – Priority

Answer 2

This is a known issue with testthat. The workaround is to add the following as the 1st line in tests/testthat.R:

Sys.setenv(R_TESTS="")

Manhood

Actually, this isn't solving the problem. Same problem as before... And still no possibility to read a log of the testthat execution :-( – Therrien

Answer 3

In case it helps someone else, this is what worked for me

1. Re-install all relevant packages. E.g. install.packages("testthat", "dplyr", "lubridate", "stringr") (I included all packages my package uses)
2. Close RStudio and reopen

Then all tests passed

Clearness

Answer 4

I spent much too long looking into this error, so hoping that I can help someone out in the future. I would like to add to this that I was getting this error while using ggplot2::autoplot() in my function and it required that I added @import ggfortify to the Roxygen skeleton part of my function.

Weems

Answer 5

I ran into the same issue with my tests failing under devtools::check() while not failing under testthat::test()

And none of the above applied to my problem, so i decided to post my issue plus solution here as well. But first some NOTEs from my experience:

devtools::check() does - so it seems - deeper error checking then your own written tests.

Now to my code-setup. I had a function that was build to retrieve values from two different files. Those files contained named profiles with a set of values per profile. But the profiles were named differently, depending on the files:

Example files:

Content of file_one:

[default]
value_A = "foo"
value_B = "bar"
value_C = "baz"

[peter]
value_A = "oof"
value_B = "rab"
value_C = "zab"

content of file_two:

[default]
value_X = "fuzzly"
value_Z = "puzzly"

[profile peter]
value_X = "fuzzly"
value_Z = "puzzly"

As you can see, does the naming in file two follow another naming convention, when it comes to the named profiles. The profiles are written in "[]" and the default-profile is always '[default]' in both files. But as soon as it comes to named profiles, its just '[name]' in one file and then '[profile name]' in the other one.

Now i've build the function like that (simplyfied):

get_value <- function(file_content, what, profile) {
  file_content <- readr::read_lines(file)
  all_profiles_at <- grep("\\[.*\\]", file_content)
  profile_regex <- paste0("\\[",if(file_content == "file_two" && profile != "default") "profile ",profile,"\\]")
  profile_at <- grep(profile_regex, file_content)
  profile_ends_at <- if(profile_at == max(all_profiles_at)) length(file_content) else all_profiles_at[grep(paste0("^",profile_at,"$"), all_profiles_at) + 1] -1
  profile_content <- file_content[profile_at:profile_ends_at]
  whole_what <- stringr::str_replace_all(profile_content[grep(paste0("^",what,".*"), profile_content)], " ", "")
  return(stringr::str_sub(whole_what, stringr::str_length(paste0(what,"=."))))
}

With this code my tests ran smoothly and even check() found no issues.

While the whole code evolved i figured, that i should read the files content beforehand and give only the alread read_in content to the function to avoid duplication in my code. So i changed the function like so:

get_value <- function(file, what, profile) {
  is_file_two <- is_file_two(file_content)
  all_profiles_at <- grep("\\[.*\\]", file_content)
  profile_regex <- paste0("\\[",if(file_content == "file_two" && profile != "default") "profile ",profile,"\\]")
  profile_at <- grep(profile_regex, file_content)
  profile_ends_at <- if(profile_at == max(all_profiles_at)) length(file_content) else all_profiles_at[grep(paste0("^",profile_at,"$"), all_profiles_at) + 1] -1
  profile_content <- file_content[profile_at:profile_ends_at]
  whole_what <- stringr::str_replace_all(profile_content[grep(paste0("^",what,".*"), profile_content)], " ", "")
  return(stringr::str_sub(whole_what, stringr::str_length(paste0(what,"=."))))
}   

As you might notice i only changed the first line of the funciton body and left the if-condition unchanged - my mistake!

But my tests didn't throw an error, as the if-condition still worked. Even though the 'file_content == "file_two"' part now generated a logical vector and if() ... else ... normally throws a warning, when the logical has length > 1. The special construct with the && doesn't throw such an error as it returns a length(1) logical:

# with warning
if(c(FALSE, FALSE, FALSE)) "Done!" else "Not done!"
# no warning:
if(c(FALSE, FALSE, FALSE) && TRUE) "Done!" else "Not done!"

Thats why my tests with testthat::test() sill worked.

But devtools::check() saw this flaw in my code and the tests failed!

And that part of the FAILURE_REPORT showed me my errors:

[...]
    where 41: test_check("my_package_name")
    
     --- value of length: 18 type: logical ---
     [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
    [13] FALSE FALSE FALSE FALSE FALSE FALSE
     --- function from context ---
[...]

Conclusion: testthat::test() is great! Is checks whether or not your code still runs. But devtools::check() goes far deeper - and when your tests pass with testthat::test() but fail with devtools::check() then you've probaly got some deeper bugs and flaws in your code you MUST attend to!

Blent

Answer 6

So as I shortly mentioned above, I changed some of my code to no longer user knitr::purl but using knitr::knit and this solved my problem.

expect_error(f <- runAllChunks('010_main_lfq_analysis.Rmd'), NA)
expect_error(f <- knitr::knit('010_main_lfq_analysis.Rmd', output='jnk.R', quiet=TRUE, envir=globalenv()), NA)

Therrien

Answer 7

This could also happen in the following scenario: You have a library already loaded in R and you are referring to the function in that library without namespace binding. For example, suppose you use the nnzero() function from the Matrix in a test file and happen to also have had the Matrix package already loaded with library(Matrix). Then devtools::test() will pass but devtools::check() fails. Using Matrix::nnzero() should fix the problem.

Lahdidah