Python, Unicode, and the Windows console

Question

When I try to print a string in a Windows console, sometimes I get an error that says UnicodeEncodeError: 'charmap' codec can't encode character ..... I assume this is because the Windows console cannot handle all Unicode characters.

How can I work around this? For example, how can I make the program display a replacement character (such as ?) instead of failing?

Faze

What version of Python are you on? I've seen references that this was broken in 2.4.3 and fixed in 2.4.4. – Capet

related: bugs.python.org/issue1602 – Milfordmilhaud

check this out. – Mcwhorter

the most simple answer that I found is to type: chcp 65001 before using pyhton in cmd – Mcwhorter

Then you should change your accepted answer... – Hallerson

The issue lies with windows default encoding, default is cp1252 you need to set it to utf8 – Brio

Answer 1

Note: This answer is sort of outdated (from 2008). Please use the solution below with care!!

---

Here is a page that details the problem and a solution (search the page for the text Wrapping sys.stdout into an instance):

PrintFails - Python Wiki

Here's a code excerpt from that page:

$ python -c 'import sys, codecs, locale; print sys.stdout.encoding; \
    sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout); \
    line = u"\u0411\n"; print type(line), len(line); \
    sys.stdout.write(line); print line'
  UTF-8
  <type 'unicode'> 2
  Б
  Б

  $ python -c 'import sys, codecs, locale; print sys.stdout.encoding; \
    sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout); \
    line = u"\u0411\n"; print type(line), len(line); \
    sys.stdout.write(line); print line' | cat
  None
  <type 'unicode'> 2
  Б
  Б

There's some more information on that page, well worth a read.

Ebsen

The link is dead and the gist of the answer wasn't quoted. -1 – Homogeneous

When I try the given advice about wrapping sys.stdout, it prints the wrong things. For example, u'\u2013' becomes û instead of an en-dash. – Hampden

@Hampden You will have to post a new question about that. Unicode and system console is not necessarily the best combination, but I don't know enough about this, so if you need a definite answer, post a question here on SO about it. – Ebsen

the link is dead. The code example is wrong for Windows console where the codepage (OEM) such as cp437 is different from Windows ANSI codepage such as cp1252. The code does not fix UnicodeEncodeError: 'charmap' codec can't encode character error and may lead to mojibake e.g., ا© is silently replaced with ╪º⌐. – Milfordmilhaud

Answer 2

Update: Python 3.6 implements PEP 528: Change Windows console encoding to UTF-8: the default console on Windows will now accept all Unicode characters. Internally, it uses the same Unicode API as the win-unicode-console package mentioned below. print(unicode_string) should just work now.

---

I get a UnicodeEncodeError: 'charmap' codec can't encode character... error.

The error means that Unicode characters that you are trying to print can't be represented using the current (chcp) console character encoding. The codepage is often 8-bit encoding such as cp437 that can represent only ~0x100 characters from ~1M Unicode characters:

>>> u"\N{EURO SIGN}".encode('cp437')
Traceback (most recent call last):
...
UnicodeEncodeError: 'charmap' codec can't encode character '\u20ac' in position 0:
character maps to 

I assume this is because the Windows console does not accept Unicode-only characters. What's the best way around this?

Windows console does accept Unicode characters and it can even display them (BMP only) if the corresponding font is configured. WriteConsoleW() API should be used as suggested in @Daira Hopwood's answer. It can be called transparently i.e., you don't need to and should not modify your scripts if you use win-unicode-console package:

T:\> py -m pip install win-unicode-console
T:\> py -m run your_script.py

See What's the deal with Python 3.4, Unicode, different languages and Windows?

Is there any way I can make Python automatically print a ? instead of failing in this situation?

If it is enough to replace all unencodable characters with ? in your case then you could set PYTHONIOENCODING envvar:

T:\> set PYTHONIOENCODING=:replace
T:\> python3 -c "print(u'[\N{EURO SIGN}]')"
[?]

In Python 3.6+, the encoding specified by PYTHONIOENCODING envvar is ignored for interactive console buffers unless PYTHONLEGACYWINDOWSIOENCODING envvar is set to a non-empty string.

Milfordmilhaud

"the default console on Windows will now accept all Unicode characters" BUT you need to configure the console: right click on the top of the windows (of the cmd or the python IDLE), in default/font choose the "Lucida console". (Japanese and Chinese don't work for me, but I should survive without it...) – Fraise

@Guillaume: the answer contains the phrase in bold about Windows console: "if the corresponding font is configured." This answer doesn't mention IDLE but you don't need to configure the font in it (I see Japanese and Chinese characters just fine in IDLE by default. Try print('\u4E01'), print('\u6b63')). – Milfordmilhaud

@Guillaume You can even get Chinese if you install the language pack in Windows 10. It added console fonts that support Chinese. – Prefect

Answer 3

Update: On Python 3.6 or later, printing Unicode strings to the console on Windows just works.

So, upgrade to recent Python and you're done. At this point I recommend using 2to3 to update your code to Python 3.x if needed, and just dropping support for Python 2.x. Note that there has been no security support for any version of Python before 3.7 (including Python 2.7) since December 2021.

If you really still need to support earlier versions of Python (including Python 2.7), you can use https://github.com/Drekin/win-unicode-console , which is based on, and uses the same APIs as the code in the answer that was previously linked here. (That link does include some information on Windows font configuration but I doubt it still applies to Windows 8 or later.)

Note: despite other plausible-sounding answers that suggest changing the code page to 65001, that did not work prior to Python 3.8. (It does kind-of work since then, but as pointed out above, you don't need to do so for Python 3.6+ anyway.) Also, changing the default encoding using sys.setdefaultencoding is (still) not a good idea.

Sleepyhead

win-unicode-console Python package (based on your code) allows to avoid modifying your script if it prints Unicode directly using py -mrun your_script.py command. – Milfordmilhaud

The answer has been updated. There is no need to include anything else from the previously linked answer. – Sleepyhead

Answer 4

If you're not interested in getting a reliable representation of the bad character(s) you might use something like this (working with python >= 2.6, including 3.x):

from __future__ import print_function
import sys

def safeprint(s):
    try:
        print(s)
    except UnicodeEncodeError:
        if sys.version_info >= (3,):
            print(s.encode('utf8').decode(sys.stdout.encoding))
        else:
            print(s.encode('utf8'))

safeprint(u"\N{EM DASH}")

The bad character(s) in the string will be converted in a representation which is printable by the Windows console.

Unifoliolate

.encode('utf8').decode(sys.stdout.encoding) leads to mojibake e.g., u"\N{EM DASH}".encode('utf-8').decode('cp437') -> ΓÇö – Milfordmilhaud

Simply print(s.encode('utf-8')) may be a better way to avoid compiler errors. Instead, you get \xNN output for unprintable characters, which was enough for my diagnostic messages. – Gourmont

This is enormously, spectacularly wrong. Encoding to UTF-8 then decoding as an 8-bit charset will a) often fail, not all codepages have characters for all 256 byte values, and b) always the wrong interpretation of the data, producing a Mojibake mess instead. – Calida

Answer 5

The below code will make Python output to console as UTF-8 even on Windows.

The console will display the characters well on Windows 7 but on Windows XP it will not display them well, but at least it will work and most important you will have a consistent output from your script on all platforms. You'll be able to redirect the output to a file.

Below code was tested with Python 2.6 on Windows.

#!/usr/bin/python
# -*- coding: UTF-8 -*-

import codecs, sys

reload(sys)
sys.setdefaultencoding('utf-8')

print sys.getdefaultencoding()

if sys.platform == 'win32':
    try:
        import win32console 
    except:
        print "Python Win32 Extensions module is required.\n You can download it from https://sourceforge.net/projects/pywin32/ (x86 and x64 builds are available)\n"
        exit(-1)
    # win32console implementation  of SetConsoleCP does not return a value
    # CP_UTF8 = 65001
    win32console.SetConsoleCP(65001)
    if (win32console.GetConsoleCP() != 65001):
        raise Exception ("Cannot set console codepage to 65001 (UTF-8)")
    win32console.SetConsoleOutputCP(65001)
    if (win32console.GetConsoleOutputCP() != 65001):
        raise Exception ("Cannot set console output codepage to 65001 (UTF-8)")

#import sys, codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)

print "This is an Е乂αmp١ȅ testing Unicode support using Arabic, Latin, Cyrillic, Greek, Hebrew and CJK code points.\n"

Buford

Is there a way to avoid this by just using a different console? – Preceptive

@sorin: Why do you first import win32console outside a try and later you do it conditionally inside a try? Isn't that kind of pointless (the first import) – Homogeneous

For what it's worth, the one provided by David-Sarah Hopwood works (I didn't get this one to even run because I haven't bothered installing the win32 extensions module) – Correspond

Don't change the system default encoding; fix your Unicode values instead. Changing the default encoding can break libraries that rely on the, you know, default behaviour. There is a reason you have to force a module reload before you can do this. – Calida

Answer 6

Just enter this code in command line before executing python script:

chcp 65001 & set PYTHONIOENCODING=utf-8

Gaylenegayler

Answer 7

Like Giampaolo Rodolà's answer, but even more dirty: I really, really intend to spend a long time (soon) understanding the whole subject of encodings and how they apply to Windoze consoles,

For the moment I just wanted sthg which would mean my program would NOT CRASH, and which I understood ... and also which didn't involve importing too many exotic modules (in particular I'm using Jython, so half the time a Python module turns out not in fact to be available).

def pr(s):
    try:
        print(s)
    except UnicodeEncodeError:
        for c in s:
            try:
                print(c, end='')
            except UnicodeEncodeError:
                print('?', end='')
                # if a logger is available (a proper one will handle any and all Unicode):
                # logger.error(f'encoding problem with character |{c}| in string |{s}|, ord(c) |{ord(c)}|, c.encode('utf-8') |{c.encode('utf-8')}|')

NB "pr" is shorter to type than "print" (and quite a bit shorter to type than "safeprint")...!

Checani

Answer 8

TL;DR:

print(yourstring.encode('ascii','replace').decode('ascii'))

---

I ran into this myself, working on a Twitch chat (IRC) bot. (Python 2.7 latest)

I wanted to parse chat messages in order to respond...

msg = s.recv(1024).decode("utf-8")

but also print them safely to the console in a human-readable format:

print(msg.encode('ascii','replace').decode('ascii'))

This corrected the issue of the bot throwing UnicodeEncodeError: 'charmap' errors and replaced the unicode characters with ?.

Pictorial

This is the solution towards I tend for my simple use case where i) the erroneous characters don't matter, and ii) I would rather have them replaced by ? than mojibake. However, you need to call decode() after your call to encode(), hence my edit. Otherwise, you get bytes instead of str. – Tobytobye

Answer 9

The cause of your problem is NOT the Win console not willing to accept Unicode (as it does this since I guess Win2k by default). It is the default system encoding. Try this code and see what it gives you:

import sys
sys.getdefaultencoding()

if it says ascii, there's your cause ;-) You have to create a file called sitecustomize.py and put it under python path (I put it under /usr/lib/python2.5/site-packages, but that is differen on Win - it is c:\python\lib\site-packages or something), with the following contents:

import sys
sys.setdefaultencoding('utf-8')

and perhaps you might want to specify the encoding in your files as well:

# -*- coding: UTF-8 -*-
import sys,time

Edit: more info can be found in excellent the Dive into Python book

Medlar

setdefaultencoding() is nolonger in sys (as of v2.0 according to the module docs). – Barbarese

I cannot prove it right now, but I know that I've used this trick on a later version - 2.5 on Windows. – Placate

OK, after quite a while I have found out that: "This function is only intended to be used by the site module implementation and, where needed, by sitecustomize. Once used by the site module, it is removed from the sys module’s namespace." – Placate

Any encoding Python outputs must be matched by the console that going to be printing it. Setting the output encoding to UTF-8 is all very well, but if the console's not UTF-8 (and on Windows it won't ever be), you'll get garbage instead of non-ASCII characters. – Mashhad

actually you can set the windows console to be utf-8. you need to say chcp 65001 and it will be unicode. – Placate

To make it absolutely clear: it is a is very a bad idea to change the default encoding. This is akin to spalking your broken leg and walking on as if nothing happened, rather than have a doctor set the bone properly. All code handling Unicode text should do so consistently instead of relying on implicit encoding / decoding. – Calida

Answer 10

Kind of related on the answer by J. F. Sebastian, but more direct.

If you are having this problem when printing to the console/terminal, then do this:

>set PYTHONIOENCODING=UTF-8

Gunflint

set PYTHONIOENCODING=UTF-8 may lead to mojibake if the console uses a different encoding such as cp437. cp65001 has various issues. To print Unicode to Windows console, Unicode API should be used (WriteConsoleW()) as suggested in my answer where PYTHONIOENCODING is used only to replace characters that can't be represented in the current OEM code page with ? (WriteConsoleW() works even for such characters). PYTHONIOENCODING can be used if the output is redirected to a file. – Milfordmilhaud

Answer 11

Python 3.6 windows7: There is several way to launch a python you could use the python console (which has a python logo on it) or the windows console (it's written cmd.exe on it).

I could not print utf8 characters in the windows console. Printing utf-8 characters throw me this error:

OSError: [winError 87] The paraneter is incorrect 
Exception ignored in: (_io-TextIOwrapper name='(stdout)' mode='w' ' encoding='utf8') 
OSError: [WinError 87] The parameter is incorrect 

After trying and failing to understand the answer above I discovered it was only a setting problem. Right click on the top of the cmd console windows, on the tab font chose lucida console.

Truly

Answer 12

For Python 2 try:

print unicode(string, 'unicode-escape')

For Python 3 try:

import os
string = "002 Could've Would've Should've"
os.system('echo ' + string)

Or try win-unicode-console:

pip install win-unicode-console
py -mrun your_script.py

Uela

Answer 13

Nowadays, the Windows console does not encounter this error, unless you redirect the output.

Here is an example Python script scratch_1.py:

s = "∞"

print(s)

If you run the script as follows, everything works as intended:

python scratch_1.py

∞

However, if you run the following, then you get the same error as in the question:

python scratch_1.py > temp.txt

Traceback (most recent call last):
  File "C:\Users\Wok\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_1.py", line 3, in <module>
    print(s)
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 0: character maps to <undefined>

---

To solve this issue with the suggestion present in the original question, i.e. by replacing the erroneous characters with question marks ?, one can proceed as follows:

s = "∞"

try:
    print(s)
except UnicodeEncodeError:
    output_str = s.encode("ascii", errors="replace").decode("ascii")

    print(output_str)

It is important:

• to call decode(), so that the type of the output is str instead of bytes,
• with the same encoding, here "ascii", to avoid the creation of mojibake.

Tobytobye

This is caused by an unrelated issue. Any file on any system can use any text encoding, but not all text encodings can represent all characters. The default is platform dependent, and is not necessarily anything to do with the terminal output. The canonical for this problem is What encoding does open() use by default?. – Subside

Answer 14

James Sulak asked,

Is there any way I can make Python automatically print a ? instead of failing in this situation?

Other solutions recommend we attempt to modify the Windows environment or replace Python's print() function. The answer below comes closer to fulfilling Sulak's request.

Under Windows 7, Python 3.5 can be made to print Unicode without throwing a UnicodeEncodeError as follows:

    In place of:    print(text)
    substitute:     print(str(text).encode('utf-8'))

Instead of throwing an exception, Python now displays unprintable Unicode characters as \xNN hex codes, e.g.:

  Halmalo n\xe2\x80\x99\xc3\xa9tait plus qu\xe2\x80\x99un point noir

Instead of

  Halmalo n’était plus qu’un point noir

Granted, the latter is preferable ceteris paribus, but otherwise the former is completely accurate for diagnostic messages. Because it displays Unicode as literal byte values the former may also assist in diagnosing encode/decode problems.

Note: The str() call above is needed because otherwise encode() causes Python to reject a Unicode character as a tuple of numbers.

Gourmont

Answer 15

The issue is with windows default encoding being set to cp1252, and need to be set to utf-8. (check PEP)

Check default encoding using:

import locale 
locale.getpreferredencoding()

You can override locale settings

import os
if os.name == "nt":
    import _locale
    _locale._gdl_bak = _locale._getdefaultlocale
    _locale._getdefaultlocale = (lambda *args: (_locale._gdl_bak()[0], 'utf8'))

referenced code from stack link

Brio