Python and Selenium To “execute_script” to solve “ElementNotVisibleException”

Question

I am using Selenium to save a webpage. The content of webpage will change once certain checkbox(s) are clicked. What I want is to click a checkbox then save the page content. (The checkboxes are controlled by JavaScript.)

Firstly I used:

driver.find_element_by_name("keywords_here").click()

it ends with an error:

NoSuchElementException

then I tried “xpath” like, with implicit/explicit waiting:

URL = “the url”

verificationErrors = []
accept_next_alert = True

aaa = driver.get(URL)
driver.maximize_window()
WebDriverWait(driver, 10)

#driver.find_element_by_xpath(".//*[contains(text(), ' keywords_here')]").click()
#Or: 

driver.find_element_by_xpath("//label[contains(text(),' keywords_here')]/../input[@type='checkbox']").click()

it gives an error:

ElementNotVisibleException

Posts

How to force Selenium WebDriver to click on element which is not currently visible?

Selenium Element not visible exception

suggest it should make the checkboxes visible before clicking, for example using:

execute_script

The question may sounds stupid, but how can I find out the proper sentence to “execute_script” the visibility of checkbox from the page source code?

Besides that, is there another way?

Thanks.

by the way, the line html code looks like:

<input type="checkbox" onclick="ComponentArt_HandleCheck(this,'p3',11);" name="keywords_here">

its xpath looks like:

//*[@id="TreeView1_item_11"]/tbody/tr/td[3]/input

Bedfellow

Answer 1

Alternative option would be to make the click() inside execute_script():

# wait for element to become present
wait = WebDriverWait(driver, 10)
checkbox = wait.until(EC.presence_of_element_located((By.NAME, "keywords_here")))

driver.execute_script("arguments[0].click();", checkbox)

where EC is imported as:

from selenium.webdriver.support import expected_conditions as EC

---

Alternatively and as an another shot in the dark, you can use the element_to_be_clickable Expected Condition and perform the click in a usual way:

wait = WebDriverWait(driver, 10)
checkbox = wait.until(EC.element_to_be_clickable((By.NAME, "keywords_here")))

checkbox.click()

Cubiculum

Sir alecxe, thanks for the help. You are an expert. I’ve read some of you posts. I tried the lines but both of them give “raise TimeoutException(message)” “selenium.common.exceptions.TimeoutException: Message: ''. – Bedfellow

@MarkK thank you. But, at the same time driver.find_element_by_xpath("//label[contains(text(),' keywords_here')]/../input[@type='checkbox']") does not produce an error? Have you checked that the xpath you use inside the find_element_by_xpath is the same used inside the "Expected Conditions" I've posted? Thanks. – Cubiculum

(sorry there's a mistake - I replaced you posted to what I had but actually I think your lines are to be added after the original lines. ) I added your lines into the originals and the xpaths in them are the same. the driver.find_element_by_xpath("//label[contains(text(),' keywords_here')]/../input[@type='checkbox']") itself gives "NoSuchElementException". – Bedfellow

@MarkK okay, thanks. Then this means we have a different problem. We are not locating the element correctly. I've updated the answer and used the By.NAME location technique. Check if we still get the NoSuchElementException. Thanks. – Cubiculum

thanks again. Does it mean only using the EC lines, or xpath lines together? (and if both EC and xpath, which goes first?) I tried only the EC By.NAME, it gives "selenium.common.exceptions.TimeoutException: Message: ". by the way, there's keyword "iframe" in the webpage's source code. is (the checkbox is in the iframe) it a concern? thanks. – Bedfellow

@MarkK oh yeah, that's a game changer. You need to switch to an iframe before looking for that checkbox element. For instance: driver.switch_to.frame("frame_id_or_name"). – Cubiculum

sir alecxe. thanks again for your replies. the problem is solved finally by using the whole xpath for its find_element_by_xpath. much appreciate your help! :) – Bedfellow

Answer 2

I had some issues with expected conditions, I prefer building my own timeout.

import time
from selenium import webdriver
from selenium.common.exceptions import \
    NoSuchElementException, \
    WebDriverException
from selenium.webdriver.common.by import By

b = webdriver.Firefox()
url = 'the url'
b.get(url)
locator_type = By.XPATH
locator = "//label[contains(text(),' keywords_here')]/../input[@type='checkbox']"
timeout = 10
success = False
wait_until = time.time() + timeout
while wait_until < time.time():
    try:
        element = b.find_element(locator_type, locator)
        assert element.is_displayed()
        assert element.is_enabled()
        element.click() # or if you really want to use an execute script for it, you can do that here.
        success = True
        break
    except (NoSuchElementException, AssertionError, WebDriverException):
        pass
if not success:
    error_message = 'Failed to click the thing!'
    print(error_message)

might want to add InvalidElementStateException, and StaleElementReferenceException

Vociferation