I have a problem with Render_site using the dopar of foreach The code works perfectly using simple for. Whereas I have a problem when using dopar since the ouput is duplicated after using dopar in the YAML file.
Note that the dopar works correctly if I use Render instead of Render_site.
Below you can see my code :
pacman::p_load(data.table)
pacman::p_load(data.table,rmarkdown)
pacman::p_load(rmarkdown,knitr,kableExtra,dplyr)
info.rmd=readLines("test.rmd",encoding = "UTF-8")
yml=readLines("site.yml",encoding = "UTF-8")
fun<-function(r){
out <- file( description=paste0(r,"test_.Rmd"), open="w", encoding="UTF-8")
write(info.rmd,file=out )
close(con=out)
render_site(paste0(r,"test_.Rmd"),encoding = "UTF-8",quiet = T)
file.remove(paste0(r,"test_.Rmd"))
}
The function is working for simple for
# Working with for
for (r in 1:10){
fun(r)
}
I have a problem using dopar (duplication of out put in the YAML file in this case)
# Not working for dopar
library(doParallel)
no_cores<-3
cl <- makeCluster(no_cores, type="SOCK")
registerDoParallel(cl)
res<-foreach (r=1:10,.combine = c,.inorder = T) %dopar% {
library(rmarkdown)
fun(r)
}
print(ret)
stopCluster(cl)
The traceback message is the following:
Error in { : task 10 failed - "Duplicate map key: 'output'"
4.stop(simpleError(msg, call = expr))
3.e$fun(obj, substitute(ex), parent.frame(), e$data)
2.foreach(r = 1:10, .combine = c, .inorder = T) %dopar% { library(rmarkdown) out <- file(description = paste0(r, "test_.Rmd"), open = "w", encoding = "UTF-8") ...
1.system.time(foreach(r = 1:10, .combine = c, .inorder = T) %dopar% { library(rmarkdown) out <- file(description = paste0(r, "test_.Rmd"), open = "w", ...
Timing stopped at: 0.39 0.36 7.9
We can then see that the _site.yml file is modified, automatically by the render_site function.
test.rmdandtest.yml? What do you mean by 'duplication of output'? What do you see if you don't render_site with the quiet option set to T? When parallel code fails mysteriously I look towards race conditions. The authors of of the packages you are using are all pretty good, so I'd be surprised, but one thing to look out for is re-use of a temporary directory relative to your working path or re-use of a temporary file relative to your working path. – Czarism