Chrome driver of selenium in headless mode not finding any element

Question

I try to make a simple selenium test in a remote server (Ubuntu 22.04) without display mode. The chrome browser version is 120.0.6099.129 and the chrome driver version is 120.0.6099.109. The result is that the selenium is not able to find any element. The error message is:

Traceback (most recent call last):

File "/home/japy/repos/myseleniumproject/main.py", line 37, in username_locator = WebDriverWait(driver,100).until(EC.presence_of_element_located((By.ID, 'fname'))) File "/home/japy/repos/myseleniumproject/python-selenium-venv/lib/python3.10/site-packages/selenium/webdriver/support/wait.py", line 105, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException

In order to get the element I tried to wait until the element is visible but then I am getting time out error. The script I am currently using is this one:

import time
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import os
import logging

log_path = './log'
logger = logging.getLogger('selenium')
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler(log_path)
logger.addHandler(handler)
logging.getLogger('selenium.webdriver.common').setLevel(logging.DEBUG)
logger.debug("this is detailed debug information:")

open(log_path, 'r')

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless=new")
driver = webdriver.Chrome(options=chrome_options)

# Go to webpage
driver.get("https://practicetestautomation.com/practice-test-login/")

#Type username student into username field
try:
    username_locator = WebDriverWait(driver,100).until(EC.presence_of_element_located((By.ID, 'username')))
finally:
    username_locator.send_keys('student')
print("username text field got")
# Type password Password12 into Password field
password_locator = driver.find_element(By.NAME, "password")
password_locator.send_keys('Password123')

# Push submit button
button = driver.find_element(By.XPATH, "//button[@class='btn']")
button.click()

The log I am getting back from the execution is the following:

this is detailed debug information: Selenium Manager binary found at: /home/japy/repos/myseleniumproject/python-selenium-venv/lib/python3.10/site-packages/selenium/webdriver/common/linux/selenium-manager Executing process: /home/japy/repos/myseleniumproject/python-selenium-venv/lib/python3.10/site-packages/selenium/webdriver/common/linux/selenium-manager --browser chrome --debug --output json chromedriver not found in PATH chrome detected at /usr/bin/google-chrome Running command: /usr/bin/google-chrome --version Output: "Google Chrome 120.0.6099.129 " Detected browser: chrome 120.0.6099.129 Required driver: chromedriver 120.0.6099.109 chromedriver 120.0.6099.109 already in the cache Driver path: /home/user/.cache/selenium/chromedriver/linux64/120.0.6099.109/chromedriver Browser path: /usr/bin/google-chrome Using driver at: /home/user/.cache/selenium/chromedriver/linux64/120.0.6099.109/chromedriver Started executable: /home/user/.cache/selenium/chromedriver/linux64/120.0.6099.109/chromedriver in a child process with pid: 3530206 using 0 to output -3 POST http://localhost:47287/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "pageLoadStrategy": "normal", "browserVersion": null, "goog:chromeOptions": {"extensions": [], "binary": "/usr/bin/google-chrome", "args": ["--headless=new", "i", "disable-gpu", "window-size=1200,1100"]}}}} Remote response: status=200 | data={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"120.0.6099.129","chrome":{"chromedriverVersion":"120.0.6099.109 (3419140ab665596f21b385ce136419fde0924272-refs/branch-heads/6099@{#1483})","userDataDir":"/tmp/.org.chromium.Chromium.5IAYJP"},"fedcm:accounts":true,"goog:chromeOptions":{"debuggerAddress":"localhost:45055"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"linux","proxy":{},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:credBlob":true,"webauthn:extension:largeBlob":true,"webauthn:extension:minPinLength":true,"webauthn:extension:prf":true,"webauthn:virtualAuthenticators":true},"sessionId":"428b77b50f61a5a83c4ff1ecc6d7371b"}} | headers=HTTPHeaderDict({'Content-Length': '850', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'}) Finished Request POST http://localhost:47287/session/428b77b50f61a5a83c4ff1ecc6d7371b/url {"url": "https://www.practicetestautomation.com/practice-test-login/"} Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'}) Finished Request GET http://localhost:47287/session/428b77b50f61a5a83c4ff1ecc6d7371b/url {} Remote response: status=200 | data={"value":"https://www.practicetestautomation.com/practice-test-login/"} | headers=HTTPHeaderDict({'Content-Length': '71', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'}) Finished Request POST http://localhost:47287/session/428b77b50f61a5a83c4ff1ecc6d7371b/element {"using": "css selector", "value": "[id="username"]"} Remote response: status=404 | data={"value":{"error":"no such element","message":"no such element: Unable to locate element: {"method":"css selector","selector":"[id="username"]"}\n (Session info: chrome=120.0.6099.129)","stacktrace":"#0 0x5579fed4ef83 \u003Cunknown>\n#1 0x5579fea07cf7 \u003Cunknown>\n#2 0x5579fea5799e \u003Cunknown>\n#3 0x5579fea57aa1 \u003Cunknown>\n#4 0x5579feaa2d64 \u003Cunknown>\n#5 0x5579fea810dd \u003Cunknown>\n#6 0x5579feaa0006 \u003Cunknown>\n#7 0x5579fea80e53 \u003Cunknown>\n#8 0x5579fea48dd4 \u003Cunknown>\n#9 0x5579fea4a1de \u003Cunknown>\n#10 0x5579fed13531 \u003Cunknown>\n#11 0x5579fed17455 \u003Cunknown>\n#12 0x5579fecfff55 \u003Cunknown>\n#13 0x5579fed180ef \u003Cunknown>\n#14 0x5579fece399f \u003Cunknown>\n#15 0x5579fed3c008 \u003Cunknown>\n#16 0x5579fed3c1d7 \u003Cunknown>\n#17 0x5579fed4e124 \u003Cunknown>\n#18 0x7f1c50e94b43 \u003Cunknown>\n"}} | headers=HTTPHeaderDict({'Content-Length': '869', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'}) Finished Request

Then the last block is repeated over and over again.

Branch

Answer 1

The problem was finally that the chrome driver was not rendering well the page due to lack of resolution setting. So in my case the problem got solved by defining the screen resolution, and together with the rest of headless settings it would look like this:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--lang=en_US")
chrome_options.add_argument("--window-size=1920,1080")
driver = webdriver.Chrome(options=chrome_options)

Branch

do you know why headless is notworking for Kayakn case? #78094100 – Proboscidean

Answer 2

I wrote a selenium script, worked fine in normal mode, but malfunctioned in headless mode. I use CDP to fetch console logs, then found many error messages:

Network error, error code: 300330

If network error occurred, script files may not be fetched, may result in elements should appear does not appear.

I tried a lot of options, but didn't work. Finally I add an option to solve this question:

chrome_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36");

Correy