Python Selenium: Can't find element by xpath when browser is headless

Question

I'm attempting to log into a website using Python Selenium using the following code:

import time
from contextlib import contextmanager
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

@contextmanager
def getBrowser(*options):
    chrome_options = Options()
    if options: [chrome_options.add_argument(option) for option in options]
    browser = webdriver.Chrome(chrome_options=chrome_options)
    try:
        yield browser
    finally:
        browser.quit()


with getBrowser() as browser:
    browser.get('https://www.vinted.com/members/notifications')

    time.sleep(20)
    browser.find_element_by_xpath('//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span').click()

It works perfectly, however when I add the --headless option to the browser, it raises a NoSuchElementException.

Error raising code:

with getBrowser('--headless') as browser:
    browser.get('https://www.vinted.com/members/notifications')

    time.sleep(20)
    browser.find_element_by_xpath('//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span').click()

Traceback:

Traceback (most recent call last):

  File "<ipython-input-4-fe0834deb137>", line 1, in <module>
    runfile('C:/Users/Alec/vinted test case.py', wdir='C:/Users/Alec')

  File "C:\Users\Alec\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "C:\Users\Alec\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/Alec/vinted test case.py", line 27, in <module>
    browser.find_element_by_xpath('//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span').click()

  File "C:\Users\Alec\selenium\webdriver\remote\webdriver.py", line 354, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)

  File "C:\Users\Alec\selenium\webdriver\remote\webdriver.py", line 832, in find_element
    'value': value})['value']

  File "C:\Users\Alec\selenium\webdriver\remote\webdriver.py", line 297, in execute
    self.error_handler.check_response(response)

  File "C:\Users\Alec\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)

NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="content"]/div/div[2]/div/div/div[6]/div[3]/div[3]/a/span"}
  (Session info: headless chrome=65.0.3325.181)
  (Driver info: chromedriver=2.36.540470 (e522d04694c7ebea4ba8821272dbef4f9b818c91),platform=Windows NT 10.0.16299 x86_64)

This error only occurs when the browser is headless. What's causing this behavior? Can it be made to work in headless mode?

target HTML:

<div class="u-flex-grow">
      <a class="c-button--inverse c-button--normal c-button--amplified c-button " href="/member/general/login?ref_url=%2Fmembers%2Fnotifications"><span class="c-button__content">Log In</span></a>
    </div>

Tarshatarshish

Try browser.find_element_by_link_text("Log In").click() – Kempis

Same issue. Doesn't work in headless, but works otherwise – Tarshatarshish

Can you update the question with the relevant HTML? – Elfont

@DebanjanB I have included the HTML of the element I am trying to interact with. Of course, the element I've provided will not have the same xpath – Tarshatarshish

Answer 1

I had once some trouble with the versions of such tools. I fixed that by ensuring I had the latest version of Chrome and webdriver installed. Maybe it's a similar issue?!

But apart of that your selector depends on a lot elements. So imagining that the html looks a bit different in headless mode could lead to your issue.

I would try to get a less strict selector like the following: //a[starts-with(@href,'/member/general/login?')]

If that is not working try to dump the html into a file from within the headless mode. Just to see what the headless browser sees and trying build a fancy selector with that.

Killick

Answer 2

I had the same issue. The solution was to add usr-agent:

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36'
chrome_options.add_argument('user-agent={0}'.format(user_agent))

I found the reason by printing the page

browser.get(url)
print(browser.page_source)

then the return html says permission deny, so I google it. Follow the solution: How to access a site via a headless driver without being denied permission

Galangal

Answer 3

I had the same issue. The solution was to provide windows size:

chrome_options.add_argument("--window-size=1920x1080")

Convalescence

Answer 4

I have the same problem when I added --headless in my python code. I resolved it using ActionChains

from selenium.webdriver import ActionChains

text_list = browser.find_elements_by_xpath("//div[@class='tab-content']")
ActionChains(browser).click(text_list).perform()

Lawful

Answer 5

I had a similar issue until I came to realize that some element classes would change depending on the theme of the website. Before realizing, I had defined my css selectors and xpaths when the page was in dark mode. When running headless, the page would be in light mode (knew by saving screenshot). I saved the whole html source and upon examining it, I found that some classes changed thus some of the css selectors and xpaths failed. What I did was to create a separate chrome profile, and from this profile I changed the site's theme to light mode. Then I redefined my css selectors and xpaths, set this profile to my bot and everything worked as it should.

Default theme of browser could be affected by your system's default theme setting

Darfur

Answer 6

Came around this issue when scrapping for tweets. Get the latest chrome user-agent for your system and provide window size.

chrome_options.add_argument('--headless')
chrome_options.add_argument("--window-size=1920x1080")
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36'
chrome_options.add_argument('user-agent={0}'.format(user_agent))

Unterwalden