could not find function inside foreach loop

Question

I'm trying to use foreach to do multicore computing in R.

A <-function(....) {
    foreach(i=1:10) %dopar% {
    B()
    }
}

then I call function A in the console. The problem is I'm calling a function Posdef inside B that is defined in another script file which I source. I had to put Posdef in the list of export argument of foreach: .export=c("Posdef"). However I get the following error:

Error in { : task 3 failed - "could not find function "Posdef""

Why cant R find this defined function?

Crackerbarrel

I would recommend searching for information regarding dynamic scoping with R. It's not at all intuitive and you'll find a lot of questions on SO that detail people having problems while defining a function within a function. – Relapse

We would appreciate a reproducible example. Also, have you tried snowfall for multicoring? I find it more intuitive and easier to adapts (modifying apply functions). – Shuman

same issue, not found any simple answer for now. I managed to make it work by calling on 4 cores foreach (j=1:N, .combine=rbind, .packages=c(VECTOR OF PACKAGES REQUIRED TO CALL ALL YOUR FUNCTIONS)) %dopar% fun(j). It improves the speed, but doesn't divide by 4 (by 2 only) I am not sure what I do so do not replicate this.. – Hankow

I solved my problem by exporting every problematic functions with clusterExport(cl, 'myMean') suggested here #17880266 – Birthplace

Answer 1

The short answer is that this was a bug in parallel backends such as doSNOW, doParallel and doMPI, but it has since been fixed.

The slightly longer answer is that foreach exports functions to the workers using a special "export" environment, not the global environment. That used to cause problems for functions that were created in the global environment, because the "export" environment wasn't in their scope, even though they were now defined in that same "export" environment. Thus, they couldn't see any other functions or variables defined in the "export" environment, such as "Posdef" in your case.

The doSNOW, doParallel and doMPI backends now change the associated environment from the global to the "export" environment for functions exported via ".export", and seems to have resolved these issues.

Windmill

I thought this issue was solved, but I am still having the same problem with the packages foreach v 1.4.4,doMPI v 0.2.2, doParallel v 1.0.14, doSNOW v 1.0.16, running on R v 3.5.0 with RStudio v 1.1.453 – Olly

Answer 2

So I can reproduce this, for the curious:

require(doSNOW)
registerDoSNOW(makeCluster(5, type="SOCK"))
getDoParWorkers()
getDoParName()
getDoParVersion()

fib <- function(n) {
  if (n <= 1) { return(1) }
  return(fib(n-1) + fib(n-2))
}

my.matrix <- matrix(runif(2500, 10, 50), nrow=50)

calcLotsaFibs <- function() {
  result <- foreach(row.num=1:nrow(my.matrix), .export=c("fib", "my.matrix")) %dopar% {
    return(Vectorize(fib)(my.matrix[row.num,]))
  }
  return(result)
}

lotsa.fibs <- calcLotsaFibs()

I have been able to get around this by putting the function in another file and loading that file in the body of the foreach. You could also obviously move the function definition into the body of the foreach itself.

[EDIT -- I had previously suggested that perhaps .export doesn't work properly with function names, but was corrected below.]

Disseisin

".export" can take function names. This is an interesting example, however, since I believe the problem is that the error occurs when fib makes a recursive call to itself, since fib is not in its own scope, as I explain in my answer. – Windmill

you mean that it could work if we wrap the custom function in a R package? – Brink

Answer 3

Quick fix for problem with foreach %dopar% is to reinstall these packages:

install.packages("doSNOW")

install.packages("doParallel") 

install.packages("doMPI")

It worked in my case.

Glenda