How to locate element in shadow-root (closed) host

Question

Its my code. I have a excel datas and wanna search these on that url but vscode cant finding elements. Why it cant find that element? I tried css selector,id, xpath but didnt work.

import openpyxl
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait

path = "dvt.xlsx"

driver = webdriver.Chrome()
url = "https://www.boschaftermarket.com/xc/en/keysecure.html"
driver.implicitly_wait(5)
driver.get(url)

wb = openpyxl.load_workbook(path)
sheet = wb.active

sayac = 1

while sayac <= 14109:
    str_getVal = "A" + str(sayac)
    # Bir sayaç başlat
    str_setval = "B" + str(sayac)

    input = driver.find_element(By.ID, "search_input")
    input.send_keys(sheet[str_getVal].value)

    button = driver.find_element(By.ID, "search_button")
    button.click()
    result = driver.find_element(
        By.XPATH, '//*[@id="rbgfCheckResponse"]/div/div/text()'
    ).text

    sheet[str_setval] = result

    sayac += 1

wb.save(path)

driver.quit()

My code giving these error. I searched more but i couldnt do it correctly. Can it be for vscode? I dont using ide btw. I dont know. Thank you.

DevTools listening on ws://127.0.0.1:61704/devtools/browser/fc1c3637-7cf4-4506-ae63-e562ce805db3       
[18836:23004:1006/092059.818:ERROR:device_event_log_impl.cc(225)] [09:20:59.820] USB: usb_service_win.cc:415 Could not read device interface GUIDs: Sistem belirtilen dosyay² bulam²yor. (0x2)
Traceback (most recent call last):
  File "c:\Users\isra0\OneDrive\Belgeler\VS Codes Projects\PythonProject\main.py", line 28, in <module>
    input = driver.find_element(By.ID, "search_input")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\isra0\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 738, in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\isra0\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 344, in execute
    self.error_handler.check_response(response)
  File "C:\Users\isra0\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="search_input"]"}
  (Session info: chrome=117.0.5938.134); For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception
Stacktrace:
        GetHandleVerifier [0x00007FF7B81B7D12+55474]
        (No symbol) [0x00007FF7B81277C2]
        (No symbol) [0x00007FF7B7FDE0EB]
        (No symbol) [0x00007FF7B801EBAC]
        (No symbol) [0x00007FF7B801ED2C]
        (No symbol) [0x00007FF7B8059F77]
        (No symbol) [0x00007FF7B803F19F]
        (No symbol) [0x00007FF7B8057EF2]
        (No symbol) [0x00007FF7B803EF33]
        (No symbol) [0x00007FF7B8013D41]
        (No symbol) [0x00007FF7B8014F84]
        GetHandleVerifier [0x00007FF7B851B762+3609346]
        GetHandleVerifier [0x00007FF7B8571A80+3962400]
        GetHandleVerifier [0x00007FF7B8569F0F+3930799]
        GetHandleVerifier [0x00007FF7B8253CA6+694342]
        (No symbol) [0x00007FF7B8132218]
        (No symbol) [0x00007FF7B812E484]
        (No symbol) [0x00007FF7B812E5B2]
        (No symbol) [0x00007FF7B811EE13]
        BaseThreadInitThunk [0x00007FFA5E7A257D+29]
        RtlUserThreadStart [0x00007FFA5FB4AA78+40]

Cottontail

Answer 1

Your element is placed in shadow-root with closed state.

When the mode of a shadow root is "closed", the shadow root's implementation internals are inaccessible and unchangeable, so you can't access shadowRoot property.

Since Selenium implemented CDP protocol API, shadow-root with closed state can be overridden with open state on page load.

You just need to override Element class prototype method attachShadow and evaluate script on new document before page load.

How to override Element properties

How to executed script on load via CDP Protocol

Example of code:

url = "https://www.boschaftermarket.com/xc/en/keysecure.html"
driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': """
Element.prototype._attachShadow = Element.prototype.attachShadow;
Element.prototype.attachShadow = function () {
    return this._attachShadow( { mode: "open" } );
};
"""})
driver.get(url)
closed_shadow_host = driver.find_element(By.CSS_SELECTOR, "aa-dxt-qr-content")
shadow_root = driver.execute_script('return arguments[0].shadowRoot', closed_shadow_host)
input = shadow_root.find_element(By.ID, "search_input")

Another option (your case specific), you can access root property which contain shadow-root structure, using JS executor.

Elements from closed shadow-root in this case are not interactable, so to interact with them you should use ActionChains.

About shadow-root

About closed state

from selenium.webdriver import ActionChains

# previous code

action = ActionChains(driver)

closed_shadow_host = driver.find_element(By.CSS_SELECTOR, "aa-dxt-qr-content")
shadow_root = driver.execute_script('return arguments[0].root.querySelector(".container-fluid")', closed_shadow_host)
input = shadow_root.find_element(By.ID, "search_input")
action.click(input).send_keys(sheet[str_getVal].value).perform()
button = shadow_root.find_element(By.ID, "search_button")
action.move_to_element(button).click().perform()

Threepence