Unable to locate elements on webpage with headless chrome

Question

I have a script that's accessing printers, and my code works totally fine when chrome is run normally, but when it's run headless, selenium can't seem to find elements on the webpage.

Here's the relevant code:

init method:

def __init__(self, ip_address):
    """ Initialize a new Printer_Webpage object."""
    self.ip_address = ip_address
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--window-size=1920x1080")
    self.browser = webdriver.Chrome(chrome_options=chrome_options)
    # Ignore lack of cert for each printer web page.
    # Otherwise, can't open page.
    self.browser.accept_untrusted_certs = True

Login method:

def login(self):
    """Navigates through the login page for the printer."""
    # Open login page
    self.browser.get(f'https://{self.ip_address}/wcd/top.xml')
    # STEPS TO LOGIN:
    # 1) Select 'Administrator' radio button and click.
    self.browser.find_element_by_id('Admin').click()
    # 2) Select Login button and click.
    self.browser.find_element_by_xpath("//input[@type='submit' \
                                        and @value='Login']").click()
    # 3) Select admin (user mode)
    self.browser.find_element_by_id('R_ADM2').click()
    # 4) Select password field and input PASSWORD, then submit.
    password_field = self.browser.find_element_by_id('Admin_Pass')
    password_field.send_keys(PASSWORD)
    password_field.send_keys(Keys.RETURN)

Full error message:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"Admin"}

And here's some other info that might be of use:

(Session info: headless chrome=62.0.3202.94)

(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 10.0.14393 x86_64)

Resurge

have the headless browser take a screenshot. It might be a matter of either scrolling to the element first or you may need to resize the headless browser when you initialize it - if it's not big enough the site may be sized for mobile and doesn't have the admin link in that view. – Parlando

Use a comma for the size of the window: --window-size=1920,1080. Take a screenshot or dump the HTML from the page to figure out why the element is not present. – Preadamite

Okay so I've found that the webpage isn't loading properly, and ends up just displaying a blank white page. When loading normally, the page is white a for a few seconds, so I tried using time.sleep(30) to give it plenty of time to load, and it still shows up as a white page. – Resurge

Chrome-headless does not work with invalid https certificates. You will only get a blank page! See bugs.chromium.org/p/chromium/issues/detail?id=721739 – Cringe

Thanks! Is there no work around for this, or will I have to switch to a different headless browser? – Resurge

Answer 1

I had the same problem. You could take screenshots to understand whats wrong.

driver.get_screenshot_as_file("screenshot.png")

A few reasons why selenium works when run normally but stops working in headless mode -

1)It might have switched to a mobile template. Can be fixed by changing the window size.

chrome_options.add_argument("--window-size=1920,1080")

2)If its a blank page (screenshot), it might be due to an invalid SSL certificate.(see @Marcel_Wilson post) It should be fixed by -

chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-running-insecure-content')

3)Its possible that the website blocks 'headless' mode. (Your screenshots might show errors which you cannot recreate in normal mode) You could try this-

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'
options.add_argument(f'user-agent={user_agent}')

However, the above code won't work if the website has a more robust blocking method. You can find more about this here https://intoli.com/blog/making-chrome-headless-undetectable/.

Invalidism

Raj spot on with Number 3 for me, I posted a more in depth answer here as there were other things to look out for as the site that made some changes to block headless also did not work properly in earlier versions of Chrome: https://mcmap.net/q/586910/-selenium-unable-to-locate-element-only-when-using-headless-chrome-python – Sello

Thank you Raj for compiling this! Works perfectly fine :) – Mathematician

adding user agent worked for me thank you. – Hilten

user agent did it for me. Thank you. Didn't see this answer anywhere else literally. – Victualler

Answer 2

I met the same situation. After studying, the following is correct：

self.chrome_options = webdriver.ChromeOptions()
self.chrome_options.add_argument("--window-size=1920,1080")
self.chrome_options.add_argument("--disable-extensions")
self.chrome_options.add_argument("--proxy-server='direct://'")
self.chrome_options.add_argument("--proxy-bypass-list=*")
self.chrome_options.add_argument("--start-maximized")
self.chrome_options.add_argument('--headless')
self.chrome_options.add_argument('--disable-gpu')
self.chrome_options.add_argument('--disable-dev-shm-usage')
self.chrome_options.add_argument('--no-sandbox')
self.chrome_options.add_argument('--ignore-certificate-errors')
self.browser = webdriver.Chrome(options=self.chrome_options)

Alfrediaalfredo

Answer 3

If it's an issue with SSL certs, you can start Chrome without certs using a command line flag (assuming that's how you're starting it). I believe the switch is --allow-running-insecure-content, and I located that from this list found here.

Bangle

Answer 4

@siking pointed out in the comments...

Chrome-headless does not work with invalid https certificates. You will only get a blank page! See http://bugs.chromium.org/p/chromium/issues/detail?id=721739

Abhor

Answer 5

I had the same problem and found that the elements were loading slower in headless. By adding the following line of code, the problem went away:

driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);

Meaghan

Answer 6

I did face the issue but the below code worked like a champ for me.

    final ChromeOptions options = new ChromeOptions();
    options.addArguments("--window-size=1920,1080");
    options.addArguments("--allow-insecure-localhost");
    options.addArguments("--headless");
    options.addArguments("--disable-gpu");
    options.addArguments("--no-sandbox");
    DesiredCapabilities caps = DesiredCapabilities.chrome();
    caps.setCapability(ChromeOptions.CAPABILITY, options);
    caps.setCapability("acceptInsecureCerts", caps);
    WebDriver driver = new ChromeDriver(options);

Impetuous

Answer 7

I had the same issue that headless doesn't work with some Websites and Elements (showing blank page content, not finding an element etc.).

This is most likely because of a missing User-Agent or a small window size. Add the following Arguments:

"--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
"--window-size=1920x1080"

In case the above doesn't work, there is also another method to "simulate" the headless mode by minimizing and moving the window to a location that can not be seen by the user.

This doesn't hide the chrome task from the taskbar, but the Chrome tab itself will still be hidden for the User.

Just use the following arguments:

var chromeOptions = new ChromeOptions();
chromeOptions.AddArguments(new List<string>() { "--window-size=1,1", "window-position=-2000,0" });  // This hides the chrome window

var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;    // This is to hid the console.

ChromeDriver driver = new ChromeDriver(chromeDriverService, chromeOptions);
//driver.Manage().Window.Minimize(); //use this if the code above does not work
driver.Navigate().GoToUrl("https://google.com");

Wichman

Answer 8

As per latest update, try using --headless=new instead of --headless. This can also be a reason if code is not working

Hern

it does not work for me. – Kitty

Answer 9

time.sleep(0.2) fixed it for me.

I had the same issue. In headless mode one webElement could not be found and selenium was throwing the NoSuchElementException with ERROR: Message: no such element: Unable to locate element.

What fixed it for me was adding a wait time of 0.2 seconds with time.sleep(0.2) the line above locating this webElement and that fixed it for me.

For some reason it needed a little more time to load this element in headless mode.

Vidal

In headless mode, rendering objects sometimes takes longer than expected. Including some wait time allows us to wait for elements to render or become visible. wait = WebDriverWait(driver, 15) element = wait.until(EC.presence_of_element_located((By.XPATH, "xpath"))) – Gilbertine