How to kill a doMC worker when it's done?

Question

The documentation for doMC seems very sparse, listing only doMC-package and registerDoMC(). The problem I'm encountering is I'll spawn several workers via doMC/foreach, but then when the job is done they just sit there taking up memory. I can go and hunt their process IDs, but I often kill the master process by accident.

library(doMC)
library(foreach)

registerDoMC(32)

foreach(i=1:32) %dopar% foo()

##kill command here?

I've tried following with registerDoSEQ() but it doesn't seem to kill off the processes.

Officeholder

possibly parallel::stopCluster might do it but I don't have any experience with doMC – Billingsley

Looking at all the dire warnings in the doMC help pages, I'd recommend using the other cluster packages instead. – Billingsley

Answer 1

I never did find a suitable solution for doMC, so for a while I've been doing the following:

library(doParallel)
cl <- makePSOCKcluster(4) # number of cores to use
registerDoParallel(cl)

## computation

stopCluster(cl)

Works every time.

Officeholder

Answer 2

The doMC package is basically a wrapper around the mclapply function, and mclapply forks workers that should exit before it returns. It doesn't use persistent workers like the snow package or the snow-derived functions in the parallel package, so it doesn't need a function like stopCluster to shutdown the workers.

Do you see the same problem when using mclapply directly? Does it work any better when you call registerDoMC with a smaller value for cores?

Are you using doMC from a IDE such as RStudio or R.app on a Mac? If so, you might want try using R from a terminal to see if that makes a difference. There could be a problem calling fork in an IDE.

Angelicangelica

Answer 3

if you using doParallel package, and using registerDoParallel(8) with numbers you can using unloadNamespace("doParallel") to kill the multi process

And if you has the name for the clusters you can using stopCluster(cl) to remove extra workers

Danieldaniela

Answer 4

By using registerDoSEQ() you simply register the sequential worker, so all parallel workers should stop. This is not a complete solution, but it should work in some cases.

Jeri