Run RSelenium in parallel

Question

How would i go about running RSelenium in parallel.

The following is an example using rvest in parallel

library(RSelenium)
library(rvest)
library(magrittr)
library(foreach)
library(doParallel)

URLsPar <- c("http://www.example.com/", "http://s5.tinypic.com/n392s6_th.jpg", "http://s5.tinypic.com/jl1jex_th.jpg",
         "http://s6.tinypic.com/16abj1s_th.jpg", "http://s6.tinypic.com/2ymvpqa_th.jpg")

(detectCores() - 1) %>%  makeCluster %>% registerDoParallel

ws <- foreach(x = 1:length(URLsPar), .packages = c("rvest", "magrittr", "RSelenium"))  %dopar%  {
      URLsPar[x] %>% read_html %>% as("character")}

stopImplicitCluster()

Cilicia

Open a separate browser for each instance using the open method of the remoteDriver class. In terms of your workflow seleniumPipes might be appropriate github.com/johndharrison/seleniumPipes – Nymphet

I have a couple of thousand urls, lets say i have 3 cores in registerDoParallel, whould i need to open 3 instances prior to the foreach ? I didn't know about the seleniumPipes! thnx – Cilicia

Answer 1

On each node in the cluster start a remoteDriver:

library(RSelenium)
library(rvest)
library(magrittr)
library(foreach)
library(doParallel)

URLsPar <- c("http://www.bbc.com/", "http://www.cnn.com", "http://www.google.com",
             "http://www.yahoo.com", "http://www.twitter.com")
appHTML <- c()
# start a Selenium Server
selServ <- startServer()

(cl <- (detectCores() - 1) %>%  makeCluster) %>% registerDoParallel
# open a remoteDriver for each node on the cluster
clusterEvalQ(cl, {
  library(RSelenium)
  remDr <- remoteDriver()
  remDr$open()
})
myTitles <- c()
ws <- foreach(x = 1:length(URLsPar), .packages = c("rvest", "magrittr", "RSelenium"))  %dopar%  {
  remDr$navigate(URLsPar[x])
  remDr$getTitle()[[1]]
}

# close browser on each node
clusterEvalQ(cl, {
  remDr$close()
})

stopImplicitCluster()
# stop Selenium Server
selServ$stop()

> ws
[[1]]
[1] "BBC - Homepage"

[[2]]
[1] "CNN - Breaking News, U.S., World, Weather, Entertainment & Video News"

[[3]]
[1] "Google"

[[4]]
[1] "Yahoo"

[[5]]
[1] "Welcome to Twitter - Login or Sign up"

Nymphet

@jdharrsion: Is it possible to open multiple tabs on a single Firefox instance, using Parallel? I am aware that environment is different in all the parallel instances, but still want to know if it is possible – Methyl

How to achieve this using docker. I have tried above code, Just inserting remoteServerAddr = "192.168.99.100", port = 4445L, browserName = "chrome"in parentheis of remDr, but it retunrs an error An unknown server-side error occurred while processing the command. – Minimalist