How to handle elements inside Shadow DOM from Selenium

Question

I want to automate file download completion checking in chromedriver. HTML of each entry in downloads list looks like

<a is="action-link" id="file-link" tabindex="0" role="link" href="http://fileSource" class="">DownloadedFile#1</a>

So I use following code to find target elements:

driver.get('chrome://downloads/')  # This page should be available for everyone who use Chrome browser
driver.find_elements_by_tag_name('a')

This returns empty list while there are 3 new downloads.

As I found out, only parent elements of #shadow-root (open) tag can be handled. So How can I find elements inside this #shadow-root element?

Aegeus

does driver.find_elements_by_id("file-link") help? – Lorenzo

no. This returns same empty list – Aegeus

okay, then probably Css/Xpath remains as the means to access driver.find_elements_by_css_selector(".[id='file-link']") provides you some value? – Lorenzo

your statement returns InvalidSelectorException, driver.find_elements_by_css_selector("[id='file-link']") returns empty list – Aegeus

@Anderson : did you miss the . after " in driver.find_elements_by_css_selector(".[id='file-link']") ? – Lorenzo

If to use . I'll get InvalidSelectorException error – Aegeus

just curious, why aren't you looking for the progressContainer or <paper-button> class? – Lorenzo

How could this help? – Aegeus

You can keep an eye on which file's progress is completed or which has a Pause/Resume button displayed against it. Does that not solve your purpose? – Lorenzo

Let us continue this discussion in chat. – Aegeus

Answer 1

Sometimes the shadow root elements are nested and the second shadow root is not visible in document root, but is available in its parent accessed shadow root. I think is better to use the selenium selectors and inject the script just to take the shadow root:

def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

outer = expand_shadow_element(driver.find_element_by_css_selector("#test_button"))
inner = outer.find_element_by_id("inner_button")
inner.click()

To put this into perspective I just added a testable example with Chrome's download page, clicking the search button needs open 3 nested shadow root elements:

import selenium
from selenium import webdriver
driver = webdriver.Chrome()


def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

driver.get("chrome://downloads")
root1 = driver.find_element_by_tag_name('downloads-manager')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_css_selector('downloads-toolbar')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_css_selector('cr-search-field')
shadow_root3 = expand_shadow_element(root3)

search_button = shadow_root3.find_element_by_css_selector("#search-button")
search_button.click()

Doing the same approach suggested in the other answers has the drawback that it hard-codes the queries, is less readable and you cannot use the intermediary selections for other actions:

search_button = driver.execute_script('return document.querySelector("downloads-manager").shadowRoot.querySelector("downloads-toolbar").shadowRoot.querySelector("cr-search-field").shadowRoot.querySelector("#search-button")')
search_button.click()

later edit:

I recently try to access the content settings(see code below) and it has more than one shadow root elements imbricated now you cannot access one without first expanding the other, when you usually have also dynamic content and more than 3 shadow elements one into another it makes impossible automation. The answer above use to work a few time ago but is enough for just one element to change position and you need to always go with inspect element an ho up the tree an see if it is in a shadow root, automation nightmare.

Not only was hard to find just the content settings due to the shadowroots and dynamic change when you find the button is not clickable at this point.

driver = webdriver.Chrome()


def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

driver.get("chrome://settings")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_css_selector('[page-name="Settings"]')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_id('search')
shadow_root3 = expand_shadow_element(root3)

search_button = shadow_root3.find_element_by_id("searchTerm")
search_button.click()

text_area = shadow_root3.find_element_by_id('searchInput')
text_area.send_keys("content settings")

root0 = shadow_root1.find_element_by_id('main')
shadow_root0_s = expand_shadow_element(root0)


root1_p = shadow_root0_s.find_element_by_css_selector('settings-basic-page')
shadow_root1_p = expand_shadow_element(root1_p)


root1_s = shadow_root1_p.find_element_by_css_selector('settings-privacy-page')
shadow_root1_s = expand_shadow_element(root1_s)

content_settings_div = shadow_root1_s.find_element_by_css_selector('#site-settings-subpage-trigger')
content_settings = content_settings_div.find_element_by_css_selector("button")
content_settings.click()

Chauchaucer

Hi Eduard I'm late to the party. I tried to use your code but it seems that shadow_root1 does not have the find_element_by_whatever method. Did I do anything wrong? Bascially I have root1 = driver.find_element_by_tag_name('input') and then shadowRoot1 = ExpandShadowElement(root1) – Frankie

They keep changing it and haven't got he time to look at it and update – Chauchaucer

Ah, thanks! Actually I found out I don't need to parse the shadow DOM, managed to log in without touching them, dunno why... – Frankie

Answer 2

There is also ready to use pyshadow pip module, which worked in my case, below example:

from pyshadow.main import Shadow
from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
shadow = Shadow(driver)
element = shadow.find_element("#Selector_level1")
element1 = shadow.find_element("#Selector_level2")
element2 = shadow.find_element("#Selector_level3")
element3 = shadow.find_element("#Selector_level4")
element4 = shadow.find_element("#Selector_level5")
element5 = shadow.find_element('#control-button') #target selector
element5.click() 

Amphictyon

I found pyshadow is only working on Chrome. In my case, it doesn't work on Firefox or Safari. – Aishaaisle

Answer 3

I would add this as a comment but I don't have enough reputation points--

The answers by Eduard Florinescu works well with the caveat that once you're inside a shadowRoot, you only have the selenium methods available that correspond to the available JS methods--mainly select by id.

To get around this I wrote a longer JS function in a python string and used native JS methods and attributes (find by id, children + indexing etc.) to get the element I ultimately needed.

You can use this method to also access shadowRoots of child elements and so on when the JS string is run using driver.execute_script()

Underskirt

Answer 4

You can use the driver.executeScript() method to access the HTML elements and JavaScript objects in your web page.

In the exemple below, executeScript will return in a Promise the Node List of all <a> elements present in the Shadow tree of element which id is host. Then you can perform you assertion test:

it( 'check shadow root content', function () 
{
    return driver.executeScript( function ()
    {
        return host.shadowRoot.querySelectorAll( 'a' ).then( function ( n ) 
        {
            return expect( n ).to.have.length( 3 )
        }
    } )
} )     

Note: I don't know Python so I've used the JavaScript syntax but it should work the same way.

Jugglery

I have no idea about what this code means :) Also I've never seen => symbol in JS What it used for?... can anyone "translate" this code? – Aegeus

() => is a lambda expression / inline function syntax. I updated my anwer to use a standard function declaration. – Jugglery

Answer 5

With selenium 4.1 there's a new attribute shadow_root for the WebElement class.

From the docs:

Returns a shadow root of the element if there is one or an error. Only works from Chromium 96 onwards. Previous versions of Chromium based browsers will throw an assertion exception.

Returns:

• ShadowRoot object or
• NoSuchShadowRoot - if no shadow root was attached to element

A ShadowRoot object has the methods find_element and find_elements but they're currently limited to:

• By.ID
• By.CSS_SELECTOR
• By.NAME
• By.CLASS_NAME

Shadow roots and explicit waits

You can also combine that with WebdriverWait and expected_conditions to obtain a decent behaviour. The only caveat is that you must use EC that accept WebElement objects. At the moment it's just one of the following ones:

• element_selection_state_to_be
• element_to_be_clickable
• element_to_be_selected
• invisibility_of_element
• staleness_of
• visibility_of

Example

e.g. borrowing the example from eduard-florinescu

from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Chrome()
timeout = 10

driver.get("chrome://settings")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = root1.shadow_root

root2 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1.find_element(by=By.CSS_SELECTOR, value='[page-name="Settings"]')))
shadow_root2 = root2.shadow_root

root3 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root2.find_element(by=By.ID, value='search')))
shadow_root3 = root3.shadow_root

search_button = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root3.find_element(by=By.ID, value="searchTerm")))
search_button.click()

text_area = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root3.find_element(by=By.ID, value='searchInput')))
text_area.send_keys("content settings")

root0 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1.find_element(by=By.ID, value='main')))
shadow_root0_s = root0.shadow_root


root1_p = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root0_s.find_element(by=By.CSS_SELECTOR, value='settings-basic-page')))
shadow_root1_p = root1_p.shadow_root


root1_s = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1_p.find_element(by=By.CSS_SELECTOR, value='settings-privacy-page')))
shadow_root1_s = root1_s.shadow_root

content_settings_div = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1_s.find_element(by=By.CSS_SELECTOR, value='#site-settings-subpage-trigger')))
content_settings = WebDriverWait(driver, timeout).until(EC.visibility_of(content_settings_div.find_element(by=By.CSS_SELECTOR, value="button")))
content_settings.click()

Moldau

Answer 6

I originally implemented Eduard's solution just slightly modified as a loop for simplicity. But when Chrome updated to 96.0.4664.45 selenium started returning a dict instead of a WebElement when calling 'return arguments[0].shadowRoot'.

I did a little hacking around and found out I could get Selenium to return a WebElement by calling return arguments[0].shadowRoot.querySelector("tag").

Here's what my final solution ended up looking like:

def get_balance_element(self):
        # Loop through nested shadow root tags
        tags = [
            "tag2",
            "tag3",
            "tag4",
            "tag5",
            ]

        root = self.driver.find_element_by_tag_name("tag1")

        for tag in tags:
            root = self.expand_shadow_element(root, tag)

        # Finally there.  GOLD!

        return [root]

def expand_shadow_element(self, element, tag):
    shadow_root = self.driver.execute_script(
        f'return arguments[0].shadowRoot.querySelector("{tag}")', element)
    return shadow_root

Clean and simple, works for me.

Also, I could only get this working Selenium 3.141.0. 4.1 has a half baked shadow DOM implementation that just manages to break everything.

Pash

Chrome 96+ is designed to work with the new shadow_dom property in Python Selenium 4.1. I also have a hack for Selenium 3 here: titusfortner.com/2021/11/22/shadow-dom-selenium.html – Dibranchiate

Answer 7

The downloaded items by google-chrome are within multiple #shadow-root (open).

---

Solution

To extract the contents of the table you have to use shadowRoot.querySelector() and you can use the following locator strategy:

• Code Block:

  driver = webdriver.Chrome(service=s, options=options)
  driver.execute("get", {'url': 'chrome://downloads/'})
  time.sleep(5)
  download = driver.execute_script("""return document.querySelector('downloads-manager').shadowRoot.querySelector('downloads-item').shadowRoot.querySelector('a#file-link')""")
  print(download.text)
  

Loftis

Answer 8

Here's a generator for getting all the web elements from the shadow DOM:

def flatten_shadows(driver):
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.wait import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC

    def get_all_elements(driver, elements):
        for el in elements:
            shadow_root = driver.execute_script('return arguments[0].shadowRoot', el)
            if shadow_root:
                shadow_els = driver.execute_script('return arguments[0].shadowRoot.querySelectorAll("*")', el)
                yield from get_all_elements(driver, shadow_els)
            else:
                yield el

    els = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '*')))
    yield from get_all_elements(driver, els)

Mcginn