Replace non-ASCII characters with a single space

Question

I need to replace all non-ASCII (\x00-\x7F) characters with a space. I'm surprised that this is not dead-easy in Python, unless I'm missing something. The following function simply removes all non-ASCII characters:

def remove_non_ascii_1(text):

    return ''.join(i for i in text if ord(i)<128)

And this one replaces non-ASCII characters with the amount of spaces as per the amount of bytes in the character code point (i.e. the – character is replaced with 3 spaces):

def remove_non_ascii_2(text):

    return re.sub(r'[^\x00-\x7F]',' ', text)

How can I replace all non-ASCII characters with a single space?

Of the myriad of similar SO questions, none address character replacement as opposed to stripping, and additionally address all non-ascii characters not a specific character.

Onesided

wow, you really took good efforts to show so many links. +1 as soon as the day renews! – Clements

You seem to have missed this one #1342500 – Li

I'm interested in seeing an example input that has problems. – Pacificas

@Stuart: Thanks, but that is the very first one that I mention. – Onesided

@dstromberg: I mention a problematic example character in the question: –. It's this guy. – Onesided

This helped me out a lot, I was having trouble while HTML parsing, but the characters that was causing a UnicodeEncodeError weren't needed, so your code just replaced with something more readable and feasible. Thanks – Selfpronouncing

If you want various somewhat better representations of the string in question, see the answers at Python - Unicode to ASCII conversion which use various useful options and charsets with string.encode(). – Sacring

... Or, to get ? instead of spaces, use something like print s.encode('ascii', 'replace') => ABRA?O JOS? for ABRAÃO JOSÉ – Sacring

Does the answer have to be in Python? Will you accept pre-processing input in something like sed, awk, or perl? – Jacky

@Jacky At this stage of the question's life, perhaps sed, awk, and perl answers would be interesting even if they are OT. But I would recommend putting them all in a single "X/Y answer", not separate answers. Usually a sed, awk, or perl answer could replace a Python answer if the code is running from e.g. a bash CLI where all four are generally available, not where actual Python scripts are running. – Onesided

Answer 1

Your ''.join() expression is filtering, removing anything non-ASCII; you could use a conditional expression instead:

return ''.join([i if ord(i) < 128 else ' ' for i in text])

This handles characters one by one and would still use one space per character replaced.

Your regular expression should just replace consecutive non-ASCII characters with a space:

re.sub(r'[^\x00-\x7F]+',' ', text)

Note the + there.

Coriolanus

@dstromberg: slower; str.join() needs a list (it'll pass over the values twice), and a generator expression will first be converted to one. Giving it a list comprehension is simply faster. See this post. – Coriolanus

The first piece of code will insert multiple blanks per character if you feed it a UTF-8 byte string. – Simons

@MarkRansom: I was assuming this to be Python 3. – Coriolanus

"– character is replaced with 3 spaces" in the question implies that the input is a bytestring (not Unicode) and therefore Python 2 is used (otherwise ''.join would fail). If OP wants a single space per Unicode codepoint then the input should be decoded into Unicode first. – Divorce

Answer 2

For you the get the most alike representation of your original string I recommend the unidecode module:

Python 2

from unidecode import unidecode
def remove_non_ascii(text):
    return unidecode(unicode(text, encoding = "utf-8"))

Then you can use it in a string:

remove_non_ascii("Ceñía")
Cenia

Python 3

from unidecode import unidecode
unidecode("Ceñía")

Longterm

interesting suggestion, but it assumes the user wishes non ascii to become what the rules for unidecode are. This however poses a follow up question to the asker about why they insist on spaces, to perhaps replace with another character? – Gerick

Thank you, this is a good answer. It doesn't work for the purpose of this question because most of the data that I'm dealing with does not have an ASCII-like representation. Such as דותן. However, in the general sense this is great, thank you! – Onesided

Yes, I know this does not work for this question, but I landed here trying to solve that problem, so I thought I’d just share my solution to my own problem, which I think is very common for people as @Onesided who deal with non-ascii characters all the time. – Longterm

There have been some security vulnerabilities with stuff like this in the past. Just be careful how you implement this! – Aristotle

Does not seem to work with UTF-16 encoded text strings – Lac

@AlvaroFuentes, how to handle/rewrite your wonderful code for Python 3 since this? Error: NameError: global name 'unicode' is not defined – Donato

This works for Python3 - if you use unidecode(text). I got some quotation marks from funny unicode characters during a crawl this way. – Farinaceous

Answer 3

For character processing, use Unicode strings:

PythonWin 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32.
>>> s='ABC马克def'
>>> import re
>>> re.sub(r'[^\x00-\x7f]',r' ',s)   # Each char is a Unicode codepoint.
'ABC  def'
>>> b = s.encode('utf8')
>>> re.sub(rb'[^\x00-\x7f]',rb' ',b) # Each char is a 3-byte UTF-8 sequence.
b'ABC      def'

But note you will still have a problem if your string contains decomposed Unicode characters (separate character and combining accent marks, for example):

>>> s = 'mañana'
>>> len(s)
6
>>> import unicodedata as ud
>>> n=ud.normalize('NFD',s)
>>> n
'mañana'
>>> len(n)
7
>>> re.sub(r'[^\x00-\x7f]',r' ',s) # single codepoint
'ma ana'
>>> re.sub(r'[^\x00-\x7f]',r' ',n) # only combining mark replaced
'man ana'

Gunman

Thank you, this is an important observation. If you do find a logical way to handle the case of combining-marks, I would happily add a bounty to the question. I suppose that simply removing the combining mark yet leaving the uncombined character alone would be best. – Onesided

A partial solution is to use ud.normalize('NFC',s) to combine marks, but not all combining combinations are represented by single codepoints. You'd need a smarter solution looking at the ud.category() of the character. – Gunman

@dotancohen: there is a notion of "user-perceived character" in Unicode that may span several Unicode codepoints. \X (eXtended grapheme cluster) regex (supported by regex module) allows to iterate over such characters (note: "graphemes are not necessarily combining character sequences, and combining character sequences are not necessarily graphemes"). – Divorce

Answer 4

If the replacement character can be '?' instead of a space, then I'd suggest result = text.encode('ascii', 'replace').decode():

"""Test the performance of different non-ASCII replacement methods."""


import re
from timeit import timeit


# 10_000 is typical in the project that I'm working on and most of the text
# is going to be non-ASCII.
text = 'Æ' * 10_000


print(timeit(
    """
result = ''.join([c if ord(c) < 128 else '?' for c in text])
    """,
    number=1000,
    globals=globals(),
))

print(timeit(
    """
result = text.encode('ascii', 'replace').decode()
    """,
    number=1000,
    globals=globals(),
))

Results:

0.7208260721400134
0.009975979187503592

Admiral

Replace the ? with a another character or space afterwards if needed, and you'd still be faster. – Ironlike

Answer 5

What about this one?

def replace_trash(unicode_string):
     for i in range(0, len(unicode_string)):
         try:
             unicode_string[i].encode("ascii")
         except:
              #means it's non-ASCII
              unicode_string=unicode_string[i].replace(" ") #replacing it with a single space
     return unicode_string

Leeke

Though this is rather inelegant, it is very readable. Thank you. – Onesided

+1 for unicode handling... @Onesided IMNSHO "readable" implies "practical" which adds to "elegant", so i'd say "a bit inelegant" – Unific

notional -1 for calling non-ascii characters "trash" – Zolly

@Zolly I meant no offense. If I recall correctly when I was writing it I was indeed dealing with very weird characters that are not from any alphabet. – Leeke

I know :) it's a light hearted comment – Zolly

Answer 6

As a native and efficient approach, you don't need to use ord or any loop over the characters. Just encode with ascii and ignore the errors.

The following will just remove the non-ascii characters:

new_string = old_string.encode('ascii',errors='ignore')

Now if you want to replace the deleted characters just do the following:

final_string = new_string + b' ' * (len(old_string) - len(new_string))

Bugbee

In python3, this encode will return a bytestring, so keep that in mind. Also, this method won't strip out characters such as newline. – Storeroom

new_string = old_string.encode('ascii', errors='ignore').decode() – Methuselah

Answer 7

When we use the ascii() it escapes the non-ascii characters and it doesn't change ascii characters correctly. So my main thought is, it doesn't change the ASCII characters, so I am iterating through the string and checking if the character is changed. If it changed then replacing it with the replacer, what you give.
For example: ' '(a single space) or '?' (with a question mark).

def remove(x, replacer):

     for i in x:
        if f"'{i}'" == ascii(i):
            pass
        else:
            x=x.replace(i,replacer)
     return x
remove('hái',' ')

Result: "h i" (with single space between).

Syntax : remove(str,non_ascii_replacer)
str = Here you will give the string you want to work with.
non_ascii_replacer = Here you will give the replacer which you want to replace all the non ASCII characters with.

Ligament

Nice edit, adding an explanation. :-) And now that I get the idea of your code I like the approach. (And as promised I did my best with formatting it for you; I hope you like it.) – Broadminded

Answer 8

Pre-processing using Raku (formerly known as Perl_6)

~$ raku -pe 's:g/ <:!ASCII>+ / /;' file

Sample Input:

Peace be upon you
السلام عليكم
שלום עליכם
Paz sobre vosotros

Sample Output:

Peace be upon you


Paz sobre vosotros

Note, you can get extensive information on the matches using the following code:

~$ raku -ne 'say s:g/ <:!ASCII>+ / /.raku;' file
$( )
$(Match.new(:orig("السلام عليكم"), :from(0), :pos(6)), Match.new(:orig("السلام عليكم"), :from(7), :pos(12)))
$(Match.new(:orig("שלום עליכם"), :from(0), :pos(4)), Match.new(:orig("שלום עליכם"), :from(5), :pos(10)))
$( )
$( )

Or more simply, you can just visualize the replacement blank spaces:

~$ raku -ne 'say S:g/ <:!ASCII>+ / /.raku;' file
"Peace be upon you"
"   "
"   "
"Paz sobre vosotros"
""

https://docs.raku.org/language/regexes#Unicode_properties
https://www.codesections.com/blog/raku-unicode/
https://raku.org

Jacky

Thank you jubilatious. I've upvoted because this is very useful knowledge for me in general, even though it is OT for this Python question. You've been very helpful with Raku / Perl questions and I appreciate that very much! – Onesided

Answer 9

def filterSpecialChars(strInput):
    result = []
    for character in strInput:
        ordVal = ord(character)
        if ordVal < 0 or ordVal > 127:
            result.append(' ')
        else:
            result.append(character)
    return ''.join(result)

And call it like this:

result = filterSpecialChars('Ceñía mañana')
print(result)

Gracegraceful

Why are you checking if ord() returns a negative number? Unicode code points are all non-negative integers, but I'll be happy to learn something new. I do agree that it is a good defensive measure, but before that I'd try to catch e.g. a TypeError exception. – Onesided

Answer 10

My problem was that my string contained things like BelgiÃ for België and &#x20AC for the € sign. And I didn't want to replace them with spaces. But wth the right symbol itself.

my solution was string.encode('Latin1').decode('utf-8')

Defence

Answer 11

Potentially for a different question, but I'm providing my version of @Alvero's answer (using unidecode). I want to do a "regular" strip on my strings, i.e. the beginning and end of my string for whitespace characters, and then replace only other whitespace characters with a "regular" space, i.e.

"Ceñíaㅤmañanaㅤㅤㅤㅤ"

to

"Ceñía mañana"

,

def safely_stripped(s: str):
    return ' '.join(
        stripped for stripped in
        (bit.strip() for bit in
         ''.join((c if unidecode(c) else ' ') for c in s).strip().split())
        if stripped)

We first replace all non-unicode spaces with a regular space (and join it back again),

''.join((c if unidecode(c) else ' ') for c in s)

And then we split that again, with python's normal split, and strip each "bit",

(bit.strip() for bit in s.split())

And lastly join those back again, but only if the string passes an if test,

' '.join(stripped for stripped in s if stripped)

And with that, safely_stripped('ㅤㅤㅤㅤCeñíaㅤmañanaㅤㅤㅤㅤ') correctly returns 'Ceñía mañana'.

Morpheme

Answer 12

To replace all non-ASCII (\x00-\x7F) characters with a space:

''.join(map(lambda x: x if ord(x) in range(0, 128) else ' ', text))

To replace all visible characters, try this:

import string

''.join(map(lambda x: x if x in string.printable and x not in string.whitespace else ' ', text))

This will give the same result:

''.join(map(lambda x: x if ord(x) in range(32, 128) else ' ', text))

Tsaritsyn