R CMD check does not respect selective code evaluation in knitr code chunks

Question

I am building a package in R 3.1.0 on Windows 7 32-bit (and also on a different machine running Windows 8 64bit) and I am using knitr to write vignettes using Markdown. I am hoping to publish this package on CRAN so I am using R CMD check to check that my demos, datasets and vignettes all behave properly. I keep my vignettes as .Rmd files (rather than making them outside of the package building process and stashing them in inst/doc) because they serve as extra tests on my package and aren't very big anyway.

My problem is that R CMD check fails when building my vignettes, even though if I run R CMD build and then R CMD INSTALL --build everything works out fine. Looking at the log file, it appears to be failing because it tries to evaluate code that I have explicitly told knitr NOT to evaluate. As a generic example, if I write

```{r example-chunk eval=c(1:3, 5:6), tidy=FALSE}
foo = 5
bar = 3
## don't evaluate the next line
file.show("i/dont/really/exist.csv")
## ok, start evaluating again
foobar = foo*bar
```

In a .Rmd file, running R CMD check will fail because it will try to evaluate line 4. However, the chunk will be correctly evaluated if I run R CMD build mypackage and then R CMD install --build mypackage.tar.gz (I know this because I can go to my Rlibs folder and find the flawless html vignettes in mypackage/doc. Similarly, the chunk will also be evaluated correctly I if run R CMD Sweave to build the vignette.

If you want to try this yourself, the package I am building (where I am running into the issue) is on Github: https://github.com/mkoohafkan/flowdurr-edu. You can look at raw/packagemaker.html for instructions, hopefully it's straightforward (the R code runs through the process of making the package directory, building the help files and copying some manually edited files into the package directory). R CMD check fails on all of my vignettes: when building flowdurr-datasets.Rmd, it insists on evaluating a line with a fake path even though I told it not to. When building hspf-analysis.Rmd, R CMD check insists on evaluating a line I excluded because it takes a really long time to complete (using rgenoud to fit some distribution parameters). R CMD check also fails on vignette.Rmd, but for a different reason; I purposely throw errors to show examples of what you can't do with a particular function, and while knitr doesn't have a problem with it R CMD check sure does!

EDIT: My build script was getting some hate so I made this dummy package that reproduced the problem on both of my machines. It illustrates that 1) R CMD check evaluates a line that it shouldn't, and 2) R CMD check does not support error evaluation in a vignette, even though knitr will write error output without issue.

So I guess my question is: What is it that R CMD check is doing differently from R CMD Sweave and R CMD build/install when it comes to vignette building, and is there anything I can do to make R CMD check respect knitr's 'eval' specification? Note that if I use eval=FALSE, R CMD check will respect it and everything is fine; the problem only occurs if I try selective evaluation of a chunk.

Viridi

Are you using R CMD check on your package source, or on the output of R CMD build? The correct sequence is R CMD check pkg; R CMD build pkg_1.0.tar.gz. I'm not sure this will make a difference, but it may do. – Superconductivity

PS. I'm curious to know why you have this complicated build script, rather than simply organising your scripts in a package. – Superconductivity

@Superconductivity interesting, I was actually under the impression that the correct sequence was to call R CMD check on the tar.gz file (I think I read that in another StackOverflow thread but I can't find it now). Thanks for the tip, unfortunately it doesn't make a difference in this case. – Viridi

@Superconductivity W.r.t. my build script, the madness in my method reflects my learning process, which was not at all straightforward. Also, the first time I ran inlinedocs (package.skeleton.dx) after running package.skeleton() it stripped all my comments out of my code, which defeated the purpose of having inline documentation. I stuck with the build script because I didn't know how to keep the documentation comments in my code. This is the first time I've ever tried to make a package, I can't help but make bad decisions ;-) – Viridi

Why not use devtools / roxygen2 for easy documentation and package building? – Peadar

@Peadar thank you for your suggestion. As I said to Andrie I know now that there are tools out there that make package building easier. I will certainly take advantage of such tools next time. My question is about knitr and R CMD Sweave though, I don't see how those packages are relevant to my specific issue. – Viridi

@Superconductivity Unfortunately you were exactly wrong -- the correct sequence is R CMD build pkg; R CMD check pkg_1.0.tar.gz (hence @Viridi is correct). The real problem here is slightly more complicated: it came from the fact that the code generated from tangle is not necessarily the code executed from weave. That is the fault of knitr at the moment, although personally I believe R CMD check should not execute the tangle output: github.com/yihui/knitr/issues/784 – Wobbling

@Yihui thanks for joining the discussion, and thanks for making such an awesome package! I agree with you that R CMD should not execute the tangle output since it is redundant. I have also found a possibly related issue when using selective evaluation in a code chunk with tidy=TRUE: If I place two comment lines in a row and selectively evaluation below the two comments, the wrong lines will be evaluated because the comments will be concatenated causing the line numbers of the remaining code to change by 1. Is that also the result of the tangle code not being the same as the weave code? – Viridi

@yihui yes, you are correct. I know that if course. But a rather spectacular way of typing too fast and thinking too slow. I apologise. – Superconductivity

@Superconductivity That is okay. It was hard for me to believe an expert like you could make such a mistake :) – Wobbling

That is a separate issue, which you can report to github.com/yihui/knitr/issues – Wobbling

Thanks @Yihui, I reported it. – Viridi

Answer 1

I have added vignette engines with the suffix _notangle in the knitr development version 1.6.2. For the original vignette engine knitr::foo, you can use the new engine knitr::foo_notangle to disable tangle (e.g. knitr::knitr_notangle, knitr::rmarkdown_notangle, ...).

If you do not want to wait for the next version of knitr to be on CRAN (which might take a while), you can certainly register a package vignette engine by yourself. Hint: you can make use of existing engines in tools::vignetteEngine(package = 'knitr') so you do not have to completely redefine the knitr vignette engines.

Wobbling

@Viridi You are welcome! I did not expect the discussion in r-devel could be so long :) – Wobbling

Answer 2

It seems that the issue is more nuanced than I originally thought, so this might not get resolved anytime soon. my workaround is to:

1. manually build the vignettes using R CMD Sweave
2. Copy the HTML outputs to inst/doc
3. Delete the vignettes folder (or add entries to .Rbuildignore--thanks @Ben!)
4. Build and check

It's not ideal, but right now it looks like the only way for my package to get through CRAN checks.

Viridi

I think you could substitute appropriate entries in a .Rbuildignore file for step #3 – Indeclinable

This is not the only way. You can take control of the vignette engines if you read the "Write R Extensions" manual. Please see my answer. – Wobbling