Convert XML/HTML Entities into Unicode String in Python [duplicate]

Question

I'm doing some web scraping and sites frequently use HTML entities to represent non ascii characters. Does Python have a utility that takes a string with HTML entities and returns a unicode type?

For example:

I get back:

ǎ

which represents an "ǎ" with a tone mark. In binary, this is represented as the 16 bit 01ce. I want to convert the html entity into the value u'\u01ce'

Povertystricken

related: Decode HTML entities in Python string? – Microcircuit

Answer 1

The standard lib’s very own HTMLParser has an undocumented function unescape() which does exactly what you think it does:

up to Python 3.4:

import HTMLParser
h = HTMLParser.HTMLParser()
h.unescape('© 2010') # u'\xa9 2010'
h.unescape('© 2010') # u'\xa9 2010'

Python 3.4+:

import html
html.unescape('© 2010') # u'\xa9 2010'
html.unescape('© 2010') # u'\xa9 2010'

Furunculosis

it also works for hex entities. The implementation is very similar to unescape() function from @dF.'s answer. – Microcircuit

This method isn't documented in Python's HTMLParser documentation, and there's a comment in the source stating it's intended for internal use. However, it works like treat in Python 2.6 through 2.7, and is probably the best solution out there. Prior to version 2.6, it would only decode named entities like & or >. – Barefaced

It is exposed as html.unescape() function in Python 3.4+ – Microcircuit

This raise UnicodeDecodeError with utf-8 strings. You must either decode('utf-8') it first or use xml.sax.saxutils.unescape. – Extine

Answer 2

Python has the htmlentitydefs module, but this doesn't include a function to unescape HTML entities.

Python developer Fredrik Lundh (author of elementtree, among other things) has such a function on his website, which works with decimal, hex and named entities:

import re, htmlentitydefs

##
# Removes HTML or XML character references and entities from a text string.
#
# @param text The HTML (or XML) source text.
# @return The plain text, as a Unicode string, if necessary.

def unescape(text):
    def fixup(m):
        text = m.group(0)
        if text[:2] == "&#":
            # character reference
            try:
                if text[:3] == "&#x":
                    return unichr(int(text[3:-1], 16))
                else:
                    return unichr(int(text[2:-1]))
            except ValueError:
                pass
        else:
            # named entity
            try:
                text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
            except KeyError:
                pass
        return text # leave as is
    return re.sub("&#?\w+;", fixup, text)

Somewhere

Absolutely. Why is not in stdlib? – No

Looking at its code, it doesn't seem to work with & and such, does it? – Clamant

Just tested successfully for & – Lobscouse

Answer 3

Use the builtin unichr -- BeautifulSoup isn't necessary:

>>> entity = '&#x01ce'
>>> unichr(int(entity[3:],16))
u'\u01ce'

Votyak

But that requires you to automatically and unambiguously know where in the string the encoded Unicode character is/are - which you can't know. And you need to try...catch the resulting exception for when you get it wrong. – No

unichar was removed in python3. Any suggestion for that version? – Unbelief

Answer 4

If you are on Python 3.4 or newer, you can simply use the html.unescape:

import html

s = html.unescape(s)

Enswathe

Answer 5

An alternative, if you have lxml:

>>> import lxml.html
>>> lxml.html.fromstring('&#x01ce').text
u'\u01ce'

Dot

Be careful though, because this can also return an object of type str if there is no special character. – Dinesh

best solution when everything fails, only lxml comes to rescue. :) – Joerg

Answer 6

You could find an answer here -- Getting international characters from a web page?

EDIT: It seems like BeautifulSoup doesn't convert entities written in hexadecimal form. It can be fixed:

import copy, re
from BeautifulSoup import BeautifulSoup

hexentityMassage = copy.copy(BeautifulSoup.MARKUP_MASSAGE)
# replace hexadecimal character reference by decimal one
hexentityMassage += [(re.compile('&#x([^;]+);'), 
                     lambda m: '&#%d;' % int(m.group(1), 16))]

def convert(html):
    return BeautifulSoup(html,
        convertEntities=BeautifulSoup.HTML_ENTITIES,
        markupMassage=hexentityMassage).contents[0].string

html = '<html>&#x01ce;&#462;</html>'
print repr(convert(html))
# u'\u01ce\u01ce'

EDIT:

unescape() function mentioned by @dF which uses htmlentitydefs standard module and unichr() might be more appropriate in this case.

Microcircuit

This solution doesn't work with the example: print BeautifulSoup('<html>&#x01ce;</html>', convertEntities=BeautifulSoup.HTML_ENTITIES) This returns the same HTML entity – Povertystricken

Note: this only applied to BeautifulSoup 3, deprecated and considered legacy since 2012. BeautifulSoup 4 handles HTML entities like these automatically. – Leon

@MartijnPieters: correct. html.unescape() is a better option on the modern Python. – Microcircuit

Absolutely. If all you wanted was to decode HTML entities there is no need to use BeatifulSoup at all. – Leon

@MartijnPieters: on old Python versions, unless HTMLParser.HTMLParser().unescape() hack worked for you, using BeautifulSoup might be a better alternative than defining unescape() by hand (vendoring a pure Python lib vs. a copy-paste of the function). – Microcircuit

Answer 7

This is a function which should help you to get it right and convert entities back to utf-8 characters.

def unescape(text):
   """Removes HTML or XML character references 
      and entities from a text string.
   @param text The HTML (or XML) source text.
   @return The plain text, as a Unicode string, if necessary.
   from Fredrik Lundh
   2008-01-03: input only unicode characters string.
   http://effbot.org/zone/re-sub.htm#unescape-html
   """
   def fixup(m):
      text = m.group(0)
      if text[:2] == "&#":
         # character reference
         try:
            if text[:3] == "&#x":
               return unichr(int(text[3:-1], 16))
            else:
               return unichr(int(text[2:-1]))
         except ValueError:
            print "Value Error"
            pass
      else:
         # named entity
         # reescape the reserved characters.
         try:
            if text[1:-1] == "amp":
               text = "&"
            elif text[1:-1] == "gt":
               text = ">"
            elif text[1:-1] == "lt":
               text = "<"
            else:
               print text[1:-1]
               text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
         except KeyError:
            print "keyerror"
            pass
      return text # leave as is
   return re.sub("&#?\w+;", fixup, text)

Elbrus

Why is this answer modded down? It seems useful to me. – Stefan

because the person wanted the character in unicode instead of utf-8 characters. I guess :) – Elbrus

Answer 8

Not sure why the Stack Overflow thread does not include the ';' in the search/replace (i.e. lambda m: '&#%d*;*') If you don't, BeautifulSoup can barf because the adjacent character can be interpreted as part of the HTML code (i.e. &#39B for &#39Blackout).

This worked better for me:

import re
from BeautifulSoup import BeautifulSoup

html_string='<a href="/cgi-bin/article.cgi?f=/c/a/2010/12/13/BA3V1GQ1CI.DTL"title="">&#x27;Blackout in a can; on some shelves despite ban</a>'

hexentityMassage = [(re.compile('&#x([^;]+);'), 
lambda m: '&#%d;' % int(m.group(1), 16))]

soup = BeautifulSoup(html_string, 
convertEntities=BeautifulSoup.HTML_ENTITIES, 
markupMassage=hexentityMassage)

1. The int(m.group(1), 16) converts the number (specified in base-16) format back to an integer.
2. m.group(0) returns the entire match, m.group(1) returns the regexp capturing group
3. Basically using markupMessage is the same as:
   html_string = re.sub('&#x([^;]+);', lambda m: '&#%d;' % int(m.group(1), 16), html_string)

Cimon

thanks for spotting the bug. I've edited my answer. – Microcircuit

Answer 9

Another solution is the builtin library xml.sax.saxutils (both for html and xml). However, it will convert only &gt, &amp and &lt.

from xml.sax.saxutils import unescape

escaped_text = unescape(text_to_escape)

Antitrust

Answer 10

Here is the Python 3 version of dF's answer:

import re
import html.entities

def unescape(text):
    """
    Removes HTML or XML character references and entities from a text string.

    :param text:    The HTML (or XML) source text.
    :return:        The plain text, as a Unicode string, if necessary.
    """
    def fixup(m):
        text = m.group(0)
        if text[:2] == "&#":
            # character reference
            try:
                if text[:3] == "&#x":
                    return chr(int(text[3:-1], 16))
                else:
                    return chr(int(text[2:-1]))
            except ValueError:
                pass
        else:
            # named entity
            try:
                text = chr(html.entities.name2codepoint[text[1:-1]])
            except KeyError:
                pass
        return text # leave as is
    return re.sub("&#?\w+;", fixup, text)

The main changes concern htmlentitydefs that is now html.entities and unichr that is now chr. See this Python 3 porting guide.

Stare

In Python 3, you'd just use html.unescape(); why have a dog and bark yourself? – Leon

html.entities.entitydefs["apos"] does not exist, and html.unescape('can't') produces "can't" which uses the U+0027 (') instead of the proper U+2019 (’) (or U+02BC, depending on which argument you follow.). But I guess that’s intended according to the character entity reference. – Luteous