- Python Parallel Wd seams to be dead from its github (last commit 9 years ago). Also it implements an obsolete protocol for selenium. Finally code is proprietary saucelabs.
Generally it's better to use SeleniumBase a Python test framework based on selenium and pytest. It's very complete and supports everything for a performance boost, parallel threads and so much more. If that's not your case ... keep reading.
Short Answer
- Both
threads and processes will give you a considerable speed up on your selenium code.
Short examples are given bellow. The selenium work is done by selenium_title function that return the page title. That don't deal with exceptions happening during each thread/process execution. For that look Long Answer - Dealing with exceptions.
- Pool of thread workers
concurrent.futures.ThreadPoolExecutor.
from selenium import webdriver
from concurrent import futures
def selenium_title(url):
wdriver = webdriver.Chrome() # chrome webdriver
wdriver.get(url)
title = wdriver.title
wdriver.quit()
return title
links = ["https://www.amazon.com", "https://www.google.com"]
with futures.ThreadPoolExecutor() as executor: # default/optimized number of threads
titles = list(executor.map(selenium_title, links))
- Pool of processes workers
concurrent.futures.ProcessPoolExecutor. Just need to replace ThreadPoolExecuter by ProcessPoolExecutor in the code above. They are both derived from the Executor base class. Also you must protect the main, like below.
if __name__ == '__main__':
with futures.ProcessPoolExecutor() as executor: # default/optimized number of processes
titles = list(executor.map(selenium_title, links))
Long Answer
Why Threads with Python GIL works?
Even tough Python has limitations on threads due the Python GIL and even though threads will be context switched. Performance gain will come due to implementation details of Selenium. Selenium works by sending commands like POST, GET (HTTP requests). Those are sent to the browser driver server. Consequently you might already know I/O bound tasks (HTTP requests) releases the GIL, so the performance gain.
Dealing with exceptions
We can make small modifications on the example above to deal with Exceptions on the threads spawned. Instead of using executor.map we use executor.submit. That will return the title wrapped on Future instances.
To access the returned title we can use future_titles[index].result where index size len(links), or simple use a for like bellow.
with futures.ThreadPoolExecutor() as executor:
future_titles = [ executor.submit(selenium_title, link) for link in links ]
for future_title, link in zip(future_titles, links):
try:
title = future_title.result() # can use `timeout` to wait max seconds for each thread
except Exception as exc: # this thread migh have had an exception
print('url {:0} generated an exception: {:1}'.format(link, exc))
Note that besides iterating over future_titles we iterate over links so in case an Exception in some thread we know which url(link) was responsible for that.
The futures.Future class are cool because they give you control on the results received from each thread. Like if it completed correctly or there was an exception and others, more about here.
Also important to mention is that futures.as_completed is better if you don´t care which order the threads return items. But since the syntax to control exceptions with that is a little ugly I omitted it here.
Performance gain and Threads
First why I've been always using threads for speeding up my selenium code:
- On I/O bound tasks my experience with selenium shows that there's minimal or no diference between using a pool of Processes (
Process) or Threads (Threads). Here also reach similar conclusions about Python threads vs processes on I/O bound tasks.
- We also know that processes use their own memory space. That means more memory consumption. Also processes are a little slower to be spawned than threads.
