I'm using beautifulsoup with html5lib, it puts the html, head and body tags automatically:
BeautifulSoup('<h1>FOO</h1>', 'html5lib') # => <html><head></head><body><h1>FOO</h1></body></html>
Is there any option that I can set, turn off this behavior ?
In [35]: import bs4 as bs
In [36]: bs.BeautifulSoup('<h1>FOO</h1>', "html.parser")
Out[36]: <h1>FOO</h1>
This parses the HTML with Python's builtin HTML parser. Quoting the docs:
Unlike html5lib, this parser makes no attempt to create a well-formed HTML document by adding a
<body>tag. Unlike lxml, it doesn’t even bother to add an<html>tag.
Alternatively, you could use the html5lib parser and just select the element after <body>:
In [61]: soup = bs.BeautifulSoup('<h1>FOO</h1>', 'html5lib')
In [62]: soup.body.next
Out[62]: <h1>FOO</h1>
<h1>a</h1><h1>b</h1> it would only return <h1>a</h1> –
Exemption soup.body.next grabs the next element in the body, as the name suggests: Is there a way to grab everything inside the body "as is" (that is, not as text)? In other words, grab the complement of soup.body.decompose() Thanks. –
Dilemma body = soup.find('body') and body.findChildren(recursive=False)[0] (I had no luck with .unwrap()) –
Dilemma children generator and join the output? "".join(map(str, soup.body.children)). –
Benjie soup.body.contents, which is a list of direct descendants (including text nodes). Or use iter(soup.body) and iterate over that. Or use list(soup.body) to produce a list of the direct descending nodes. They are all the same thing, really; list() and iter() call the __iter__ method, and the __iter__ method returns iter(self.contents). –
Benjie soup.children in this case. Unless you're parsing a full document you won't have a body –
Exemption <html>, <head> and <body> tag have been added. That's because BeautifulSoup deals with whole documents, not with fragments, and html5lib (and lxml too, by the way) repair broken HTML documents by adding the required but missing elements back in. –
Benjie This aspect of BeautifulSoup has always annoyed the hell out of me.
Here's how I deal with it:
# Parse the initial html-formatted string
soup = BeautifulSoup(html, 'lxml')
# Do stuff here
# Extract a string repr of the parse html object, without the <html> or <body> tags
html = "".join([str(x) for x in soup.body.children])
A quick breakdown:
# Iterator object of all tags within the <body> tag (your html before parsing)
soup.body.children
# Turn each element into a string object, rather than a BS4.Tag object
# Note: inclusive of html tags
str(x)
# Get a List of all html nodes as string objects
[str(x) for x in soup.body.children]
# Join all the string objects together to recreate your original html
"".join()
I still don't like this, but it gets the job done. I always run into this when I use BS4 to filter certain elements and/or attributes from HTML documents before doing something else with them where I need the entire object back as a string repr rather than a BS4 parsed object.
Hopefully, the next time I Google this, I'll find my answer here.
decode_contents() now, see my answer below, which is aimed to stop the annoyance I also had before. –
Keller html and body tags are added to it, by default. WRT to the str vs BS4 parse object; did you feel my post didn't draw attention to that appropriately at the bottom? –
Sikorsky Let's first create a soup sample:
soup=BeautifulSoup("<head></head><body><p>content</p></body>")
You could get html and body's child by specify soup.body.<tag>:
# python3: get body's first child
print(next(soup.body.children))
# if first child's tag is rss
print(soup.body.rss)
Also you could use unwrap() to remove body, head, and html
soup.html.body.unwrap()
if soup.html.select('> head'):
soup.html.head.unwrap()
soup.html.unwrap()
If you load xml file, bs4.diagnose(data) will tell you to use lxml-xml, which will not wrap your soup with html+body
>>> BS('<foo>xxx</foo>', 'lxml-xml')
<foo>xxx</foo>
soup.html.unwrap() then soup.body.unwrap() worked for me! –
Beaumarchais <H1> and <h1> are the same thing, but in XML, where tag names are case-sensitive, they are not. So <H1>Foo</h1> is valid HTML, but not valid XML. –
Benjie You may have misunderstood BeautifulSoup here. BeautifulSoup deals with whole HTML documents, not with HTML fragments. What you see is by design.
Without a <html> and <body> tag, your HTML document is broken. BeautifulSoup leaves it to the specific parser to repair such a document, and different parsers differ in how much they can repair. html5lib is the most thorough of the parsers, but you'll get similar results with the lxml parser (but lxml leaves out the <head> tag). The html.parser parser is the least capable, it can do some repair work but it doesn't add back required but missing tags.
So this is a deliberate feature of the html5lib library, it fixes HTML that is lacking, such as adding back in missing required elements.
There is not option for BeautifulSoup to treat the HTML you pass in as a fragment. At most you can 'break' the document and remove the <html> and <body> elements again with the standard BeautifulSoup tree manipulation methods.
E.g. using Element.replace_with() lets you replace the html element with your <h1> element:
>>> soup = BeautifulSoup('<h1>FOO</h1>', 'html5lib')
>>> soup
<html><head></head><body><h1>FOO</h1></body></html>
>>> soup.html.replace_with(soup.body.contents[0])
<html><head></head><body></body></html>
>>> soup
<h1>FOO</h1>
Take into account however, that html5lib can add other elements to your tree too, such as tbody elements:
>>> BeautifulSoup(
... '<table><tr><td>Foo</td><td>Bar</td></tr></table>', 'html5lib'
... ).table
<table><tbody><tr><td>Foo</td><td>Bar</td></tr></tbody></table>
The HTML standard states that a table should always have a <tbody> element, and if it is missing, a parser should treat the document as if the element is there anyway. html5lib follows the standard very, very closely.
html5lib expects complete HTML documents, not HTML fragments. So whatever the data was meant to be, html5lib is not the library to use for HTML fragments. Hell, BeautifulSoup is not the library to use, a BeautifulSoup object expects to handle complete documents, but it's up to the parser to repair things if not. –
Benjie Yet another solution:
from bs4 import BeautifulSoup
soup = BeautifulSoup('<p>Hello <a href="http://google.com">Google</a></p><p>Hi!</p>', 'lxml')
# content handling example (just for example)
# replace Google with StackOverflow
for a in soup.findAll('a'):
a['href'] = 'http://stackoverflow.com/'
a.string = 'StackOverflow'
print ''.join([unicode(i) for i in soup.html.body.findChildren(recursive=False)])
html=str(soup)
html=html.replace("<html><body>","")
html=html.replace("</body></html>","")
will remove the html/body tag bracket. A more sophisticated version would also check for startsWith, endsWith ...
str(soup.body.contents[0]) (for single-element bodies, or use looping and str.join()) or even simpler, soup.body.decode_contents()? The latter is, unfortunately, not documented but is the method that Element.decode() calls to render the contents when rendering itself, and Element.decode() is the base implementation for __str__. –
Benjie If you want it to look better, try this:
BeautifulSoup([contents you want to analyze].prettify())
Since v4.0.1 there's a method decode_contents():
>>> BeautifulSoup('<h1>FOO</h1>', 'html5lib').body.decode_contents()
'<h1>FOO</h1>'
More details in a solution to this question: https://mcmap.net/q/88849/-beautifulsoup-innerhtml
Update:
As rightfully noted by @MartijnPieters in the comments this way you'll still get some extra tags like tbody (in the tables) which you might or might not want.
tbody tag, you'll find that html5lib has added one; '<table><tr><td>Foo</td></tr></table>' -> '<html><head></head><body><table><tbody><tr><td>Foo</td></tr></tbody></table></body></html>', and that might be unexpected too. –
Benjie tbody. –
Keller tbody tags. "Maybe they wanted a BeautifulSoup object without those tags?" - or maybe not. We do not know for sure. In the meanwhile maybe another reader is interested in strings particularly. –
Keller Here is how I do it
a = BeautifulSoup()
a.append(a.new_tag('section'))
#this will give you <section></section>
© 2022 - 2024 — McMap. All rights reserved.
innerHTMLdoes), then you want a different API. – Hardener