make python replace un-encodable chars with a string by default

Question

I want to make python ignore chars it can't encode, by simply replacing them with the string "<could not encode>".

E.g, assuming the default encoding is ascii, the command

'%s is the word'%'ébác'

would yield

'<could not encode>b<could not encode>c is the word'

Is there any way to make this the default behavior, across all my project?

Novelize

If the default encoding is ascii, what encoding is that 'ébác' string in? – Kalman

@Peter Hansen - you're right :) it was just to explain what I want... bad example. – Novelize

see also set the implicit default encoding\decoding error handling in python – Baur

Answer 1

The str.encode function takes an optional argument defining the error handling:

str.encode([encoding[, errors]])

From the docs:

Return an encoded version of the string. Default encoding is the current default string encoding. errors may be given to set a different error handling scheme. The default for errors is 'strict', meaning that encoding errors raise a UnicodeError. Other possible values are 'ignore', 'replace', 'xmlcharrefreplace', 'backslashreplace' and any other name registered via codecs.register_error(), see section Codec Base Classes. For a list of possible encodings, see section Standard Encodings.

In your case, the codecs.register_error function might be of interest.

[Note about bad chars]

By the way, note when using register_error that you'll likely find yourself replacing not just individual bad characters but groups of consecutive bad characters with your string, unless you pay attention. You get one call to the error handler per run of bad chars, not per char.

Gausman

There are examples for how to use codecs.register_error in this Python test file. – Dumbbell

@NadavB Thx! Here’s a fresh link. – Dumbbell

Answer 2

>>> help("".encode)
Help on built-in function encode:

encode(...)
S.encode([encoding[,errors]]) -> object

Encodes S using the codec registered for encoding. encoding defaults
to the default encoding. errors may be given to set a different error
handling scheme. Default is 'strict' meaning that encoding errors raise
a UnicodeEncodeError. **Other possible values are** 'ignore', **'replace'** and
'xmlcharrefreplace' as well as any other name registered with
codecs.register_error that is able to handle UnicodeEncodeErrors.

So, for instance:

>>> x
'\xc3\xa9b\xc3\xa1c is the word'
>>> x.decode("ascii")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
>>> x.decode("ascii", "replace")
u'\ufffd\ufffdb\ufffd\ufffdc is the word'

Add your own callback to codecs.register_error to replace with the string of your choice.

Bituminous

Answer 3

minimal example for codecs.register_error
based on this answer (which is more verbose)

#!/usr/bin/env python3

import codecs

def some_handler(exception):
    return (b"-", exception.end)

codecs.register_error("some_handler", some_handler)

s = 'A\uff1aB' # \uff1a = Fullwidth Colon
_bytes = s.encode("latin1", errors="some_handler")
print(repr(_bytes)) # b'A-B'

Baur