Convert from hex character to Unicode character in python

Question

The hex string '\xd3' can also be represented as: Ó.

The easiest way I've found to get the character representation of the hex string to the console is:

print unichr(ord('\xd3'))

Or in English, convert the hex string to a number, then convert that number to a unicode code point, then finally output that to the screen. This seems like an extra step. Is there an easier way?

Chavira

Answer 1

print u'\xd3'

Is all you have to do. You just need to somehow tell Python it's a unicode literal; the leading u does that. It will even work for multiple characters.

If you aren't talking about a literal, but a variable:

codepoints = '\xd3\xd3'
print codepoints.decode("latin-1")

Edit: Specifying a specific encoding when printing will not work if it's incompatible with your terminal encoding, so just let print do encode(sys.stdout.encoding) automatically. Thanks @ThomasK.

Seamark

But, what do you do if you have a string composed of code points? How would you convert the string to the actual string representation? Is u a function? Can we write u(some_string)? – Seventh

@Geo: u is not a function, it's part of the string literal syntax. But what is a 'string composed of code points'? – Kirbie

@agf: OK, great. btw, I think print u'\xd3'.encode("latin-1") is wrong - on a UTF-8 terminal, that shows nothing at all. – Kirbie

What is latin-1? I was stuck because I was trying .encode("utf-8") and getting an error. – Chavira

@ThomasK I can't check because I think my terminal is latin-1 or that silly, almost the same winodws encoding. It worked-for-me. – Seamark

@Kevin latin-1 is another ascii compatible encoding. It's similar to cp-whatever that is a common windows encoding so it often works when utf-8 fails because you're not actually using utf-8. You should read docs.python.org/howto/unicode.html. \xd3 doesn't seem to be valid by itself in utf-8, but part of a two byte sequence. print u'\xd3'.encode("utf-8") gives two characters. – Seamark

@agf: latin-1 or c1252 (almost the same) is a fallback for decoding, because it can accept any byte. It doesn't work the other way, because it will only encode certain unicode characters. UTF-8 will encode any character. For printing cross platform, try .encode(sys.stdout.encoding) (or just print the unicode - CPython at least does the same thing inside print). – Kirbie

It’s a bit better in Python3 (Python2 is being sunsetted) but Unicode and Python do not get along very well even on the best of days, and you cannot use Python’s re library for Unicode per UTS#18’s level-1 reqs. Matthew Barnett’s regex library for both Python2 and Python3 helps a lot. See the chart on Slide 6 of my Unicode Support Shootout talk to see how dramaticaly better MRAB’s regex lib is than normal Python re. Alas,you still have Python’s fatal multiple personality disorder on UCS‐²⁄₄ though. – Iverson

Moderator Note Several comments under this answer were removed because they provided more noise than signal. Please try to keep comments constructive and on topic, related to the answer at hand. If you want to have a 'side chat', use the chat system. – Assorted

Answer 2

if data is something like this "\xe0\xa4\xb9\xe0\xa5\x88\xe0\xa4\xb2\xe0\xa5\x8b \xe0\xa4\x95\xe0\xa4\xb2"

sys.stdout.buffer.write(data) 

would print

हैलो कल

Yautia

This comment deserves to be a question. This resolved my problem. – Marozas

Answer 3

Not long ago, I had a very similar problem. I had to decode files that contained unicode hex (e.g., _x0023_) instead of special characters (e.g., #). The solution is described in the following code:

Script

from collections import OrderedDict
import re


def decode_hex_unicode_to_latin1(string: str) -> str:
    hex_unicodes = list(OrderedDict.fromkeys(re.findall(r'_x[?:\da-zA-Z]{4}_', string)))

    for code in hex_unicodes:
        char = bytes.fromhex(code[2:-1]).decode("latin1")[-1]
        string = string.replace(code, char)

    return string


def main() -> None:
    string = "|_x0020_C_x00f3_digo_x0020_|"
    decoded_string = decode_hex_unicode_to_latin1(string)
    print(string, "-->", decoded_string)

    return


if __name__ == '__main__':
    main()

Output

|_x0020_C_x00f3_digo_x0020_| --> | Código |

Oldfangled

I think you can re-write that loop with just decoded_string = re.sub(r'_x([a-fA-F0-9]{4})_', lambda x: bytes.fromhex(x.group(1)).decode("utf-8"), string). – Bebel