chromedriver suddenly slow (scraping with python, selenium)

Question

Have a python script running some scrapers using selenium and chromedriver.

Have been scraping the same sites for a few years now with no issues. Starting last night, the same sites have started to load EXTREMELY slowly when loaded through chromedriver, though loading on my regular un-automated browser is totally fine. I've tried uninstalling and reinstalling chromedriver, upgrading, restarting, etc. to no avail. This has happened across two completely separate sites, both became slow starting last night. I am not blocked from the sites, but they load much slower than anything else.

It almost feels like a memory allocation issue, as even javascript and scrolling performs much slower than it used to. But I changed no code, and the issue arose even without an update to chromedriver (used to be fast on 112, but then became slow last night while still on version 112).

Using Selenium 4.2.0, and ChromeDriver 113.0.5672.63, though I was on version 112 yesterday and still seeing the error.

Does anyone know if there was a widespread change or something I'm not aware of?

Olva

Selenium 4.2.0 is a bit old. Have you tried 4.9.0? – Gifford

@Gifford just upgraded to 4.9.0. everything runs fine, but the issue still remains. painfully slow and unable to really process anything in a reasonable way. – Olva

Not sure how much this might help your particular situtation but headless browsers and drivers can make your scripts faster. – Gifford

@Gifford I think due to the angular dynamic loading and other anti-scraping measures it's not really possible with headless – Olva

I've been seeing similar behavior. I've noticed chrome doesn't feel as snappy as it used to. This started within the last week or so. That said, I have a few chrome_options (from years of bug workarounds) that I have yet to eliminate to see if any are causing the issue with the latest build. – Penile

I have a test suite that takes 7 minutes in chrome but only 4 minutes in edge (older version of chromium) using identical browser options. – Penile

I have the same issue with 4.9.1 – Inhumane

I'm having the same issue. I'm on Selenium 4.4.3 and the newest version of Chrome (114.0.5735.133) and the newest version of Chromedriver (114.0.5735.90). I see that Chromedriver is launching the wrong architecture but like you I cannot find a way to fix it. I tried upgrading to Selenium 4.10 but it didn't make a difference so I downgraded back. I need to be able to run some of these scrapers in headed mode as well. Hopefully we can figure out a solution soon. – Plummet

Answer 1

it happening because chromedriver started version with wrong architecture, in my case chromedriver started version x86_x64 when my Mac using arm64 architecture

I don't know how to fix it, maybe someone knows

uninstall the current JDK version and install the latest arm version from oracle.com/java/technologies/downloads/#jdk20-mac (without usage brew)

Enchilada

Hi I've tried this, I'm running JDK 20, but it's still not working. Still doing the x86 version. Have you got any workarounds? – Rianna

definitely seems like the likely culprit here - would love to understand how to properly resolve in a maintainable way – Olva

I've been using rosetta in my terminal, so brew install chromedriver downloaded the x86_64 one.. once I realized that, I just uninstalled and installed again in an arm terminal and it worked! Would never have guessed without your answer thx! – Harass

Even after uninstalling current JDK and installing the latest, I'm getting the same issue. Any other pointers please? – Ode

Answer 2

Have you tried downgrading your chrome? I use webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options) to auto-download chrome drivers on my script based on my chrome version. I checked that my chrome was updated 2 days ago and my script became slow to even load pages. My chrome updated to new stable 113.0.5672.92/93 and as I downgraded to an old stable version, all my scripts started working normally as before. Use following guide to downgrade:

https://helpdeskgeek.com/how-to/how-to-downgrade-google-chrome-to-an-older-version/

If this does not work, wait for few days to get official release of new stable chrome_driver which might solve the issue.

Hungry

Answer 3

I also had this issue with my Amazon Corretto 11 which Intellij downloaded automatically on my ARM architecture. When printing System.getProperty("os.arch") I was getting x86_x64 instead of aarm64.

I've removed corretto and installed OpenJDK11 version and issue is fixed.

Sanctuary

Answer 4

I had this same issue. Turned out that the inappropriate version of python was installed (see this discussion - my chromedriver version was correct).

Since I'm using anaconda to run python via jupyter notebooks, I first created a new environment "Native" with the new version of python, I followed the guidance from here.

CONDA_SUBDIR=osx-arm64 conda create -n native numpy -c conda-forge

will get you a osx-arm64 native env.

To make it permanent, do,

conda activate native conda config --env --set subdir osx-arm64

Then I opened Jupyter notebook from within the "native" environment and chromedriver was back to being fast.

Ardenia