Login/Register
Decode HTML entities in Python string?
Asked Answered
Solved pythonhtmlhtml-entities
Y

7

382

I'm parsing some HTML with Beautiful Soup 3, but it contains HTML entities which Beautiful Soup 3 doesn't automatically decode for me:

>>> from BeautifulSoup import BeautifulSoup

>>> soup = BeautifulSoup("<p>&pound;682m</p>")
>>> text = soup.find("p").string

>>> print text
&pound;682m

How can I decode the HTML entities in text to get "£682m" instead of "&pound;682m".

Yarvis answered 18/1, 2010 at 16:8 Comment(1)
related: Convert XML/HTML Entities into Unicode String in Python – Bullet
B
726

Python 3.4+

Use html.unescape():

import html
print(html.unescape('&pound;682m'))

FYI html.parser.HTMLParser.unescape is deprecated, and was supposed to be removed in 3.5, although it was left in by mistake. It will be removed from the language soon.

Python 2.6-3.3

You can use HTMLParser.unescape() from the standard library:

>>> try:
...     # Python 2.6-2.7 
...     from HTMLParser import HTMLParser
... except ImportError:
...     # Python 3
...     from html.parser import HTMLParser
... 
>>> h = HTMLParser()
>>> print(h.unescape('&pound;682m'))
£682m

You can also use the six compatibility library to simplify the import:

>>> from six.moves.html_parser import HTMLParser
>>> h = HTMLParser()
>>> print(h.unescape('&pound;682m'))
£682m
Bonneau answered 18/1, 2010 at 16:17 Comment(7)
this method doesn't seem to escape characters like "&#8217;" on google app engine, though it works locally on python2.6. It does still decode entities (like &quot;) at least – Spenserian
How can an undocumented API be deprecated? Edited the answer. – Badgett
@MarkusUnterwaditzer there's no reason that an undocumented method can't be deprecated. This one throws deprecation warnings - see my edit to the answer. – Moor
It would seem more logical that, rather than just the unescape method, the entire HTMLParser module were deprecated in favor of html.parser. – Constantin
Worth noting for Python 2: Special characters are replaced with their Latin-1 (ISO-8859-1) encoding counterparts. E.g., it may be necessary to h.unescape(s).encode("utf-8"). The docs: """The definition provided here contains all the entities defined by XHTML 1.0 that can be handled using simple textual substitution in the Latin-1 character set (ISO-8859-1)""" – Clitoris
It does not work for 'Don&‌#039;t forget that &‌pi; = 3.14 &‌amp; doesn&‌#039;t equal 3.' WHY is that? – Verena
@Verena Those work for me. html.unescape('Don&#039;t forget that &pi; = 3.14 &amp; doesn&#039;t equal 3.') produces "Don't forget that π = 3.14 & doesn't equal 3." for me on Python 3.11.3 – Ruhnke
V
72

Beautiful Soup handles entity conversion. In Beautiful Soup 3, you'll need to specify the convertEntities argument to the BeautifulSoup constructor (see the 'Entity Conversion' section of the archived docs). In Beautiful Soup 4, entities get decoded automatically.

Beautiful Soup 3

>>> from BeautifulSoup import BeautifulSoup
>>> BeautifulSoup("<p>&pound;682m</p>", 
...               convertEntities=BeautifulSoup.HTML_ENTITIES)
<p>£682m</p>

Beautiful Soup 4

>>> from bs4 import BeautifulSoup
>>> BeautifulSoup("<p>&pound;682m</p>")
<html><body><p>£682m</p></body></html>
Vulgate answered 18/1, 2010 at 16:19 Comment(4)
+1. No idea how I missed this in the docs: thanks for the info. I'm going to accept luc's answer tho because his uses the standard lib which I specified in the question (not important to me) and its probably of more general use to other people. – Yarvis
BeautifulSoup4 uses HTMLParser, mostly. See the source – Naturalism
How do we get the conversion in Beautiful Soup 4 without all the extraneous HTML that wasn't part of the original string? (i.e. <html> and <body>) – Vachill
@Vachill : BeautifulSoup('&pound;682m', "html.parser") https://mcmap.net/q/88170/-don-39-t-put-html-head-and-body-tags-automatically-beautifulsoup – Coefficient
S
16

You can use replace_entities from w3lib.html library

In [202]: from w3lib.html import replace_entities

In [203]: replace_entities("&pound;682m")
Out[203]: u'\xa3682m'

In [204]: print replace_entities("&pound;682m")
£682m
Samuella answered 9/8, 2016 at 8:52 Comment(0)
W
8

Beautiful Soup 4 allows you to set a formatter to your output

If you pass in formatter=None, Beautiful Soup will not modify strings at all on output. This is the fastest option, but it may lead to Beautiful Soup generating invalid HTML/XML, as in these examples:

print(soup.prettify(formatter=None))
# <html>
#  <body>
#   <p>
#    Il a dit <<Sacré bleu!>>
#   </p>
#  </body>
# </html>

link_soup = BeautifulSoup('<a href="http://example.com/?foo=val1&bar=val2">A link</a>')
print(link_soup.a.encode(formatter=None))
# <a href="http://example.com/?foo=val1&bar=val2">A link</a>
Wept answered 14/1, 2014 at 10:3 Comment(2)
This doesn't answer the question. (Also, I have no idea what the docs are saying is invalid about the final bit of HTML here.) – Moor
<<Sacré bleu!>> is the invalid part, as it has unescaped < and > and will break the html around it. I know this is a late post from me, but in case anyone happens to be looking and wondered... – Plaudit
S
1

I had a similar encoding issue. I used the normalize() method. I was getting a Unicode error using the pandas .to_html() method when exporting my data frame to an .html file in another directory. I ended up doing this and it worked...

    import unicodedata

The dataframe object can be whatever you like, let's call it table...

    table = pd.DataFrame(data,columns=['Name','Team','OVR / POT'])
    table.index+= 1

encode table data so that we can export it to out .html file in templates folder(this can be whatever location you wish :))

     #this is where the magic happens
     html_data=unicodedata.normalize('NFKD',table.to_html()).encode('ascii','ignore')

export normalized string to html file

    file = open("templates/home.html","w") 

    file.write(html_data) 

    file.close()

Reference: unicodedata documentation

Shotgun answered 2/4, 2020 at 21:3 Comment(1)
This does not answer the question. A Unicode normalization form is not the same as HTML entities. – Eiderdown
C
0
import html
  
myHtml = "<body><h1> How to use html.unescape() in Python </h1></body>"
encodedHtml = html.escape(myHtml)
print("Encoded HTML: ", encodedHtml)
decodedHtml = html.unescape(encodedHtml)
  
print("Decoded HTML: ", decodedHtml)

Output:

Encoded HTML:  &lt;body&gt;&lt;h1&gt; How to use html.unescape() in Python &lt;/h1&gt;&lt;/body&gt;
Decoded HTML:  <body><h1> How to use html.unescape() in Python </h1></body>

Demo

Cubature answered 17/8, 2023 at 18:43 Comment(0)
F
-5

This probably isnt relevant here. But to eliminate these html entites from an entire document, you can do something like this: (Assume document = page and please forgive the sloppy code, but if you have ideas as to how to make it better, Im all ears - Im new to this).

import re
import HTMLParser

regexp = "&.+?;" 
list_of_html = re.findall(regexp, page) #finds all html entites in page
for e in list_of_html:
    h = HTMLParser.HTMLParser()
    unescaped = h.unescape(e) #finds the unescaped value of the html entity
    page = page.replace(e, unescaped) #replaces html entity with unescaped value
Furtherance answered 18/12, 2012 at 18:28 Comment(1)
No! You don't need to match HTML entities yourself and loop over them; .unescape() does that for you. I don't understand why you and Rob have posted these overcomplicated solutions that roll their own entity matching when the accepted answer already clearly shows that .unescape() can find entities in the string. – Moor

© 2022 - 2024 — McMap. All rights reserved.