How to setup testthat for R CMD check?

Question

There are apparently two ways to integrate testthat with R CMD check. I can't get either to work.

Approach #1: (perhaps deprecated)

According to the devtools wiki:

When developing a package, put your tests in inst/tests and then create a file tests/run-all.R (note that it must be a capital R), which contains the following code:

library(testthat) 
library(mypackage)
test_package("mypackage") 

This will evaluate your tests in the package namespace (so you can test non-exported functions), and it will throw an error if there are any test failures. This means you'll see the full report of test failures and R CMD check won't pass unless all tests pass.

The whole package is here. In it are the two files:

## minimalbugexample/inst/tests/run-all.R
library(testthat)
library(minimalbugexample)
test_package('minimalbugexample')

and

## minimalbugexample/inst/tests/test-use-Matrix-package.R
context("Intentional break")
  expect_that( TRUE, equals(FALSE))

my DESCRIPTION is

Package: minimalbugexample
Title: 
Description: 
Version: 0.1.1
Author: Nathan VanHoudnos <[email protected]>
Maintainer: Nathan VanHoudnos <[email protected]>
Depends:
    R (>= 3.0.1),
    Matrix (>= 1.0)
Suggests:
    testthat
License: GPL
LazyData: true
Collate:
    'minimalbugexample-package.r'
    'use-Matrix-package.R'

After installing the package, I can run the tests just fine (they fail, as expected).

> test_package('minimalbugexample')
Intentional break : 1


1. Failure:  -------------------------------------------------------------------
TRUE not equal to FALSE
1 element mismatch
Error: Test failures
> 

But an R CMD check doesn't run the tests.

$ R CMD check minimalbugexample_0.1.1.tar.gz 
... snip ...
* checking PDF version of manual ... WARNING
WARNING: There was 1 warning.
See
  ‘/home/nathanvan/software/minimalbugexample.Rcheck/00check.log’
for details.

I don't think that the PDF warning has anything to do with this, but I can provide more details if requested.

Approach #2: (bleeding edge)

According to the README file of the testthat repository:

Now, recommend practice is to put your tests in tests/testthat, and ensure R CMD check runs then by putting the following code in tests/test-all.R:

library(testthat)
test_check(yourpackage)

So I made sure I had the most recent version of testthat installed:

> install_github("testthat")

And then changed the package. You can get this version here. I modified the two files to be

## minimalbugexample/inst/tests/test-all.R
library(testthat)
test_check(minimalbugexample)

and

## minimalbugexample/inst/tests/testthat/test-use-Matrix-package.R
context("Intentional break")
  expect_that( TRUE, equals(FALSE))

Then updating the package version to 0.1.2 in the DESCRIPTION file, I can build it, install it, and use testthat to check it and get the same output as before. So it seems that as far as testthat is concerned, its working.

However, R CMD check still doesn't run the tests:

$ R CMD check minimalbugexample_0.1.2.tar.gz 
... snip ...
* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
WARNING: There was 1 warning.
See
  ‘/home/nathanvan/software/minimalbugexample.Rcheck/00check.log’
for details.

So the question:

What am I doing wrong? My preference is for a solution for Approach 2, but I'll take either!

Abdel

Just a note that if you're using approach #1 you need to add testthat to Suggests: in DESCRIPTION. – Paganize

Answer 1

You don't have a tests directory. test-all.R should be located at minimalbugexample/tests/test-all.R.

Then your actual tests go in minimalbugexample/inst/tests for approach #1 or minimalbugexample/tests/testthat/ for approach #2.

For approach #2, the test-all.R file should use test_check(yourpackage) instead of test_package(yourpackage) and the library(yourpackage) call is no longer required.

Baseline

I guess the issue is that I can't read! Thanks! (Works for Approach #2!) – Abdel

Actually, for approach #2, the test-all.R file should be located at minimalbugexample/tests/test-all.R as you say, but the actual tests go in minimalbugexample/tests/testthat/. It took some more bug testing for me to work it out. If you incorporate that into your answer I'll re-accept it! Example on gitHub. – Abdel

Thanks. Editing is a new priv for me, so I was unsure of the community norms. – Abdel

Also, to avoid a warning about unspecified dependencies, you should add Suggests: testthat to your DESCRIPTION file – Chamorro

Running the tests is amongst the last actions of R CMD check <pkg>, so any errors that occur earlier could prevent them from running. Such as the lack of a NAMESPACE file. – Enow

Also note that the uppercase extension to test-all.R is essential. Otherwise R CMD check won't run the tests. – Marlea