Python ElementTree won't convert non-breaking spaces when using UTF-8 for output

Asked 18/5, 2012 at 13:40 Answered 29/5, 2012 at 3:37

I'm trying to parse, manipulate, and output HTML using Python's ElementTree:

import sys
from cStringIO  import StringIO
from xml.etree  import ElementTree as ET
from htmlentitydefs import entitydefs

source = StringIO("""<html>
<body>
<p>Less than &lt;</p>
<p>Non-breaking space &nbsp;</p>
</body>
</html>""")

parser = ET.XMLParser()
parser.parser.UseForeignDTD(True)
parser.entity.update(entitydefs)
etree = ET.ElementTree()

tree = etree.parse(source, parser=parser)
for p in tree.findall('.//p'):
    print ET.tostring(p, encoding='UTF-8')

When I run this using Python 2.7 on Mac OS X 10.6, I get:

<p>Less than &lt;</p>

Traceback (most recent call last):
  File "bar.py", line 20, in <module>
    print ET.tostring(p, encoding='utf-8')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1120, in tostring
    ElementTree(element).write(file, encoding, method=method)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 815, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 931, in _serialize_xml
    write(_escape_cdata(text, encoding))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1067, in _escape_cdata
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 19: ordinal not in range(128)

I thought that specifying "encoding='UTF-8'" would take care of the non-breaking space character, but apparently it doesn't. What should I do instead?

Wellthoughtof answered 18/5, 2012 at 13:40 Comment(0)

0xA0 is a latin1 character, not a unicode character and the value of p.text in the loop is a str and not unicode, that means that in order to encode it in utf-8 it must first be converted by Python implicitly into a unicode string (i.e. using decode). When it is doing this it assumes ascii since it wasn't told anything else. 0xa0 is not a valid ascii character, but it is a valid latin1 character.

The reason you have latin1 characters instead of unicode characters is because entitydefs is a mapping of names to latin1 encode strings. You need the unicode code point which you can get from htmlentitydef.name2codepoint

The version below should fix it for you:

import sys
from cStringIO  import StringIO
from xml.etree  import ElementTree as ET
from htmlentitydefs import name2codepoint

source = StringIO("""<html>
<body>
<p>Less than &lt;</p>
<p>Non-breaking space &nbsp;</p>
</body>
</html>""")

parser = ET.XMLParser()
parser.parser.UseForeignDTD(True)
parser.entity.update((x, unichr(i)) for x, i in name2codepoint.iteritems())
etree = ET.ElementTree()

tree = etree.parse(source, parser=parser)
for p in tree.findall('.//p'):
    print ET.tostring(p, encoding='UTF-8')

Gavrilla answered 29/5, 2012 at 2:37 Comment(1)

This is the correct answer! To put it more concisely, htmlentitydefs.entitydefs is bad. It’s causing byte strings to be added into your ElementTree where there should only be unicode strings. And unfortunately the error doesn’t surface until later. – Horseradish 29/5, 2012 at 18:33

XML only defines <, >, ', " and &.   and others come from HTML. So you have a couple of choices.

You can change your source to use numeric entities, like   or   both of which are equivalent to  .
You can use a DTD which defines those values.

There is some useful information (it is written about XSLT, but XSLT is written using XML, so the same applies) at the XSLT FAQ.

The question appears now to include a stack trace; that changes things. Are you sure that the string is in UTF-8? If it resolves to the single byte 0xA0, then it isn't UTF-8 but more likely cp1252 or iso-8859-1.

Groscr answered 18/5, 2012 at 13:54 Comment(1)

The problem isn't on input: the UseForeignDTD trick works fine for that. The problem is on output: the text in memory contains 0xA0, which I expected would be converted to its UTF-8 representation by ET.tostring (since I said 'encoding="UTF-8"'). – Wellthoughtof 20/5, 2012 at 1:55

Your   is being converted to '\xa0' which is the default (ascii) encoding for a nonbreaking space (the UTF-8 encoding is '\xc2\xa0'.) The line

'\xa0'.encode('utf-8')

results in a UnicodeDecodeError, because the default codec, ascii, only works up to 128 characters and ord('\xa0') = 160. Setting the default encoding to something else, i.e.:

import sys
reload(sys)  # must reload sys to use 'setdefaultencoding'
sys.setdefaultencoding('latin-1')

print '\xa0'.encode('utf-8', "xmlcharrefreplace")

should solve your problem.

Oringas answered 29/5, 2012 at 3:37 Comment(0)

-1

HTML is not the same as XML, so tags like   will not work. Ideally, if you are trying to pass that information via XML, you could first xml-encode the above data, so it would look something like this:

<xml>
<mydata>
&lt;htm&gt;
&lt;body&gt;
&lt;p&gt;Less than &amp;lt;&lt;/p&gt;
&lt;p&gt;Non-breaking space &amp;nbsp;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;
</mydata>
</xml>

And then after parsing the XML you can HTML-unencode the string.

Breeks answered 18/5, 2012 at 14:9 Comment(1)

-1

I think the problem you have here is not with your nbsp entity but with your print statement.

Your error is:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 19: ordinal not in range(128)

I think this is because you're taking a utf-8 string (from ET.tostring(p, encoding='utf-8')) and trying to echo it out in a ascii terminal. So Python is implicitly converting that string to unicode then converting it again to ascii. Although nbsp can be represented directly in utf-8, it cannot be represented directly in ascii. Hence the error.

Try saving the output to a file instead and seeing if you get what you expect.

Alternatively, try print ET.toString(p, encoding='ascii'), which should cause ElementTree to use numeric character entities to represent anything that can't be represented with ascii.

Crevice answered 18/5, 2012 at 15:5 Comment(1)

Saving the output to a file has no effect: if I open a file using "output = open('temp.txt', 'w')" and then use "output.write(ET.tostring(p, encoding='ascii'))", I get the same error. – Wellthoughtof 20/5, 2012 at 1:58

Recommended topics

Hot tags