S

19

663

I'm using the Python bindings to run Selenium WebDriver:

from selenium import webdriver
wd = webdriver.Firefox()

I know I can grab a webelement like so:

elem = wd.find_element_by_css_selector('#my-id')

And I know I can get the full page source with...

wd.page_source

But is there a way to get the "element source"?

elem.source   # <-- returns the HTML as a string

The Selenium WebDriver documentation for Python are basically non-existent and I don't see anything in the code that seems to enable that functionality.

What is the best way to access the HTML of an element (and its children)?

Sanderson answered 31/8, 2011 at 21:44 Comment(2)

You also could just parse all the wd.page_source with beautifulsoup – Antonina 1/3, 2013 at 13:59

selenium has more facilities compare to beautiful soup. – Haldes 19/5, 2021 at 10:38

G

1015

You can read the innerHTML attribute to get the source of the content of the element or outerHTML for the source with the current element.

Python:

element.get_attribute('innerHTML')

Java:

elem.getAttribute("innerHTML");

C#:

element.GetAttribute("innerHTML");

Ruby:

element.attribute("innerHTML")

JavaScript:

element.getAttribute('innerHTML');

PHP:

$element->getAttribute('innerHTML');

It was tested and worked with the ChromeDriver.

Globoid answered 20/12, 2011 at 12:49 Comment(11)

innerHTML is a not DOM attribute. So above answer wouldn't work. innerHTML is a javascript javascript value. Doing above would return null. The answer by nilesh is the proper answer. – Delimitate 22/3, 2012 at 13:53

This works great for me, and is much more elegant than the accepted answer. I'm using Selenium 2.24.1. – Flavoring 10/7, 2012 at 2:4

Though innerHTML is not a DOM attribute, it is well supported by all major browsers (quirksmode.org/dom/w3c_html.html). It works also well for me. – Triolet 23/7, 2012 at 10:57

+1 This appears to work in ruby also. I have a feeling that the getAttribute method (or equivalent in other languages) just calls the js method whose name is the arg. However the documentation doesn't explicitly say this, so nilesh's solution should be a fallback. – Montemontefiascone 20/8, 2012 at 19:45

I'm getting this: content.get_attribute('innerHTML') == u'<div>...</div>' – Breughel 30/8, 2012 at 20:0

This fails for HtmlUnitDriver. Works for ChromeDriver, FirefoxDriver, InternetExplorerDriver (IE10) and PhantomJSDriver (I haven't tested others). – Chetchetah 22/5, 2014 at 20:54

@Chetchetah - HtmlUnit's javascript support is pretty weak; I'd imagine by extension they haven't supported this. More info at this thread – Riel 17/6, 2014 at 20:12

In Ruby it element.attribute("innerHTML") if anyone needs it. – Unsustainable 7/11, 2014 at 20:36

nice- elem.get_attribute("innerHTML") It works by using this. – Masker 26/4, 2016 at 12:43

in Node: element.getAttribute('innerHTML') – Additory 10/10, 2016 at 19:36

I have a situation where an alert text is displayed and I want to catch. Neither driver.page_source nor element.get_attribute('innerHTML') can return contents containing that alert text. In Firefox,only "inspect elements |right click <HTML>|copy|inner HTML" give me what I need. How can I mimic that action chain in Selenium Python? – Handiness 26/3, 2020 at 22:10

B

103

There is not really a straightforward way of getting the HTML source code of a webelement. You will have to use JavaScript. I am not too sure about python bindings, but you can easily do like this in Java. I am sure there must be something similar to JavascriptExecutor class in Python.

 WebElement element = driver.findElement(By.id("foo"));
 String contents = (String)((JavascriptExecutor)driver).executeScript("return arguments[0].innerHTML;", element);

Bryna answered 3/9, 2011 at 3:29 Comment(5)

This is essentially what I ended up doing, albeit with the Python equivalent. – Sanderson 7/9, 2011 at 21:15

I think the answer below, using element.getAttribute("innerHTML") is a lot easier to read. I don't understand why people are voting it down. – Flavoring 10/7, 2012 at 2:5

No need to call javascript at all. In Python just use element.get_attribute('innerHTML') – Misbeliever 30/4, 2014 at 8:15

@Misbeliever innerHTMLis not a DOM attribute. When I answered this question in 2011, it did not work for me, looks like now some browsers are supporting it. If it works for you then using innerHTML is cleaner. However there is no guarantee it will work on all browsers. – Bryna 30/4, 2014 at 13:25

Apparently, this is the only way to get innerHTML while using RemoteWebDriver – Lesser 6/6, 2015 at 14:29

P

100

Here's how to get the HTML source code using Selenium Python:

elem = driver.find_element("xpath", "//*")
source_code = elem.get_attribute("outerHTML")

Here's how to save that HTML to a file:

with open('c:/html_source_code.html', 'w') as f:
    f.write(source_code.encode('utf-8'))

Peltast answered 20/3, 2013 at 18:8 Comment(4)

Can I set a delay and get the latest source? There are dynamic contents loaded using javascript. – Piazza 17/10, 2013 at 23:41

Does this work even if the page is not fully loaded? Also, is there any way to set a delay like @FlyingAtom mentioned? – Ligule 20/10, 2014 at 16:1

If Webpage contain dynamic contents then it depends upon behavior of that webpage but 90%, you had to set delay before getting raw HTML from that page. And most simplest way is time.sleep(x) # Where x is seconds to set delay. – Attenuate 4/1, 2021 at 12:25

This is an old answer. Nowadays, the method find_element_by_xpath no longer exists, and this gives AttributeError: 'WebDriver' object has no attribute 'find_element_by_xpath'. So, now, instead of driver.find_element_by_xpath("//*"), you should use driver .find_element("xpath", "//*"). Found that in this answer. – Thailand 14/9, 2022 at 4:12

C

15

In Ruby, using selenium-webdriver (2.32.1), there is a page_source method that contains the entire page source.

Cataplasia answered 15/4, 2013 at 20:59 Comment(0)

V

9

The other answers provide a lot of details about retrieving the markup of a WebElement. However, an important aspect is, modern websites are increasingly implementing JavaScript, ReactJS, jQuery, Ajax, Vue.js, Ember.js, GWT, etc. to render the dynamic elements within the DOM tree. Hence there is a necessity to wait for the element and its children to completely render before retrieving the markup.

Python

Hence, ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using get_attribute("outerHTML"):

element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#my-id")))
print(element.get_attribute("outerHTML"))

Using execute_script():

element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#my-id")))
print(driver.execute_script("return arguments[0].outerHTML;", element))

Note: You have to add the following imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Veritable answered 27/11, 2020 at 12:31 Comment(1)

An essential question is, what kind of HTML I get: 1) just the source tunneled through selenium or b) the source after chrome (depending on the driver also Safari or Firefox) rendered it? – Amoeboid 1/5, 2023 at 9:34

C

7

It looks outdated, but let it be here anyway. The correct way to do it in your case:

elem = wd.find_element_by_css_selector('#my-id')
html = wd.execute_script("return arguments[0].innerHTML;", elem)

or

html = elem.get_attribute('innerHTML')

Both are working for me (selenium-server-standalone-2.35.0).

Chally answered 6/3, 2014 at 14:52 Comment(0)

D

6

Using the attribute method is, in fact, easier and more straightforward.

Using Ruby with the Selenium and PageObject gems, to get the class associated with a certain element, the line would be element.attribute(Class).

The same concept applies if you wanted to get other attributes tied to the element. For example, if I wanted the string of an element, element.attribute(String).

Diego answered 22/3, 2013 at 15:46 Comment(0)

G

4

Java with Selenium 2.53.0

driver.getPageSource();

Grayish answered 29/3, 2016 at 21:25 Comment(3)

that's not what the question asked for – Densitometer 31/5, 2017 at 2:18

Depending on the webdriver, the getPageSource method may not return the actual page source (ie with possible javascript changements). The returned source may be the raw source sent by the server. The webdriver doc must be checked to ensure this point. – Informal 25/7, 2017 at 6:23

Also works for php - $driver->getPageSource() – Emee 21/12, 2021 at 18:45

T

4

InnerHTML will return the element inside the selected element and outerHTML will return the inside HTML along with the element you have selected

Example:

Now suppose your Element is as below

<tr id="myRow"><td>A</td><td>B</td></tr>

innerHTML element output

<td>A</td><td>B</td>

outerHTML element output

<tr id="myRow"><td>A</td><td>B</td></tr>

Live Example:

http://www.java2s.com/Tutorials/JavascriptDemo/f/find_out_the_difference_between_innerhtml_and_outerhtml_in_javascript_example.htm

Below you will find the syntax which require as per different binding. Change the innerHTML to outerHTML as per required.

Python:

element.get_attribute('innerHTML')

Java:

elem.getAttribute("innerHTML");

If you want whole page HTML, use the below code:

driver.getPageSource();

Tamarin answered 3/9, 2017 at 7:18 Comment(0)

F

3

Updated 2022 Selenium Retrieving HTML

To start with, download the Python bindings for Selenium WebDriver.

One can do this from the PyPI page for the Selenium package.
Alternatively, one can use pip to install the Selenium package. Python 3.6 provides the pip in the standard library.

Method 1

Read the innerHTML attribute to get the source of the element’s content. innerHTML is a property of a DOM element whose value is the HTML between the opening tag and ending tag.

For example, the innerHTML property in the code below carries the value “text”

<p>
a text
</p>

element.get_attribute('innerHTML')

Method 2

Read the outerHTML to get the source with the current element. outerHTML is an element property whose value is the HTML between the opening and closing tags and the HTML of the selected element itself.

For example, the code’s outerHTML property carries a value that contains div and span inside that.

<div>
<span>Hello there!</span>
</div>

ele.get_atrribute("outerHTML")

Fortieth answered 15/12, 2022 at 10:51 Comment(0)

B

2

I hope this could help: http://selenium.googlecode.com/svn/trunk/docs/api/java/org/openqa/selenium/WebElement.html

Here is described Java method:

java.lang.String    getText()

But unfortunately it's not available in Python. So you can translate the method names to Python from Java and try another logic using present methods without getting the whole page source...

E.g.

 my_id = elem[0].get_attribute('my-id')

Battle answered 7/9, 2011 at 14:23 Comment(3)

Python actually does have a "gettext" equivalent (I think its just the "text" attribute?) but that actually just returns the "plaintext" between HTML tags and won't actually return the full HTML source. – Sanderson 7/9, 2011 at 21:17

This returns only the plain text (not the html) in Java too. – Flavoring 10/7, 2012 at 2:6

you must reference it like you said elem[0] otherwise it doesn't work – Dvina 12/9, 2013 at 18:17

G

2

The method to get the rendered HTML I prefer is the following:

driver.get("http://www.google.com")
body_html = driver.find_element_by_xpath("/html/body")
print body_html.text

However, the above method removes all the tags (yes, the nested tags as well) and returns only text content. If you interested in getting the HTML markup as well, then use the method below.

print body_html.getAttribute("innerHTML")

Grenville answered 4/2, 2018 at 17:32 Comment(2)

You can also use driver.find_element_by_tag("body") to reach the body content of the page. – Grenville 5/2, 2018 at 4:58

This works in Excel VBA with Selenium but needs some adjustment. – Chaldron 8/3, 2023 at 11:6

A

2

This works seamlessly for me.

element.get_attribute('innerHTML')

Aruspex answered 22/9, 2019 at 15:26 Comment(0)

A

0

If you are interested in a solution for Selenium Remote Control in Python, here is how to get innerHTML:

innerHTML = sel.get_eval("window.document.getElementById('prodid').innerHTML")

Angelangela answered 9/7, 2013 at 14:18 Comment(1)

Thanks for the help, I have used this. I also find innerHTML = {solenium selector code}.text works just the same. – Coldshoulder 4/8, 2013 at 0:1

C

0

Use execute_script get html

bs4(BeautifulSoup) also can access html tag quickly.

from bs4 import BeautifulSoup
html = adriver.execute_script("return document.documentElement.outerHTML")
bs4_onepage_object=BeautifulSoup(html,"html.parser")
bs4_div_object=bs4_onepage_object.find_all("atag",class_="attribute")

Citronella answered 11/9, 2021 at 2:49 Comment(0)

S

0

In current versions of php-webdriver (1.12.0+) you have to use

$element->getDomProperty('innerHTML');

as pointed out in this issue: https://github.com/php-webdriver/php-webdriver/issues/929

Seow answered 25/10, 2021 at 12:10 Comment(2)

Why an answer using PHP when the question is specifically about Python? – Conlan 4/2, 2023 at 4:18

@Conlan I answered you above. The reason is that for similar requests for PHP, Google issues this page – Emee 8/2, 2023 at 6:27

E

0

In PHP Selenium WebDriver you can get page source like this:

$html = $driver->getPageSource();

Or get HTML of the element like this:

// innerHTML if you need HTML of the element content
$html = $element->getDomProperty('outerHTML');

Emee answered 22/12, 2021 at 7:51 Comment(3)

Question is about Python not PHP – Conlan 4/2, 2023 at 4:18

@Conlan I know i can read but google search for php returns this page – Emee 5/2, 2023 at 6:33

@Conlan this answer has upvotes, which means it was helpful to someone – Emee 5/2, 2023 at 6:35

L

-1

WebElement element = driver.findElement(By.id("foo"));
String contents = (String)((JavascriptExecutor)driver).executeScript("return arguments[0].innerHTML;", element);

This code really works to get JavaScript from source as well!

Laicize answered 31/8, 2012 at 4:4 Comment(0)

F

-1

And in PHPUnit Selenium test it's like this:

$text = $this->byCssSelector('.some-class-nmae')->attribute('innerHTML');

Foy answered 30/5, 2014 at 10:25 Comment(1)

Question is about Python not PHP – Conlan 4/2, 2023 at 4:19