UnicodeEncodeError: 'ascii' codec can't encode characters

Question

I have a dict that's feed with url response. Like:

>>> d
{
0: {'data': u'<p>found "\u62c9\u67cf \u591a\u516c \u56ed"</p>'}
1: {'data': u'<p>some other data</p>'}
...
}

While using xml.etree.ElementTree function on this data values (d[0]['data']) I get the most famous error message:

UnicodeEncodeError: 'ascii' codec can't encode characters...

What should I do to this Unicode string to make it suitable for ElementTree parser?

PS. Please don't send me links with Unicode & Python explanation. I read it all already unfortunately, and can't make use of it, as hopefully others can.

Gastroenterostomy

Answer 1

You'll have to encode it manually, to UTF-8:

ElementTree.fromstring(d[0]['data'].encode('utf-8'))

as the API only takes encoded bytes as input. UTF-8 is a good default for such data.

It'll be able to decode to unicode again from there:

>>> from xml.etree import ElementTree
>>> p = ElementTree.fromstring(u'<p>found "\u62c9\u67cf \u591a\u516c \u56ed"</p>'.encode('utf8'))
>>> p.text
u'found "\u62c9\u67cf \u591a\u516c \u56ed"'
>>> print p.text
found "拉柏 多公 园"

Deprived

Yes, that was the first thing I tried and I always try. Problem is with ElementTree.tostring. Can you please try ElementTree.tostring(p, method='text') and tell why it doesn't work? Thanks – Gastroenterostomy

Ah, sorry. It was too obvious. .tostring() has optional argument 'encoding' which is probably set to ascii by default, so adding encoding='utf-8' works. Cheers – Gastroenterostomy

@theta: Hehe, just about to tell you about that. :-) – Deprived

for more information, check this issue: bugs.python.org/issue11033 – Viticulture