Detect strings with non English characters in Python

Question

I have some strings that have a mix of English and none English letters. For example:

w='_1991_اف_جي2'

How can I recognize these types of string using Regex or any other fast method in Python?

I prefer not to compare letters of the string one by one with a list of letters, but to do this in one shot and quickly.

Scalariform

maybe use the ascii range since ascii os only english characters in the range of 0-255 i believe – Dispatcher

Can you tell me how to do this in Python? – Scalariform

@Scalariform Which Python version are you using? – Catherin

Checkout this answer. Don't forget to upvote that answer and the question :) – Canadianism

Answer 1

You can just check whether the string can be encoded only with ASCII characters (which are Latin alphabet + some other characters). If it can not be encoded, then it has the characters from some other alphabet.

Note the comment # -*- coding: ..... It should be there at the top of the python file (otherwise you would receive some error about encoding)

# -*- coding: utf-8 -*-
def isEnglish(s):
    try:
        s.encode(encoding='utf-8').decode('ascii')
    except UnicodeDecodeError:
        return False
    else:
        return True

assert not isEnglish('slabiky, ale liší se podle významu')
assert isEnglish('English')
assert not isEnglish('ގެ ފުރަތަމަ ދެ އަކުރު ކަ')
assert not isEnglish('how about this one : 通 asfަ')
assert isEnglish('?fd4))45s&')

Mountie

Thanks for the answer. In Python 3 what you said was not working correctly, buy I used what you suggested and replaced s.decode('ascii') with s.encode('ascii') and also UnicodeDecodeError` with UnicodeEnecodeError and then it worked. – Scalariform

I was indeed using Python2 to test my code. Thanks for improving the solution for python3 – Mountie

I edited this answer to work with both python 2 and 3. – Aphonic

@TJ1, your approach is correct for Python3, you just have a typo - it's UnicodeEncodeError – Lantz

Answer 2

IMHO it is the simpliest solution:

def isEnglish(s):
  return s.isascii()

print(isEnglish("Test"))
print(isEnglish("_1991_اف_جي2"))

Output:
True
False

Banking

isascii was introduced in python 3.7. so minimum requirement to use this function you must have >= python 3.7 – Bristling

Answer 3

If you work with strings (not unicode objects), you can clean it with translation and check with isalnum(), which is better than to throw Exceptions:

import string

def isEnglish(s):
    return s.translate(None, string.punctuation).isalnum()


print isEnglish('slabiky, ale liší se podle významu')
print isEnglish('English')
print isEnglish('ގެ ފުރަތަމަ ދެ އަކުރު ކަ')
print isEnglish('how about this one : 通 asfަ')
print isEnglish('?fd4))45s&')
print isEnglish('Текст на русском')

> False
> True
> False
> False
> True
> False

Also you can filter non-ascii characters from string with this function:

ascii = set(string.printable)   

def remove_non_ascii(s):
    return filter(lambda x: x in ascii, s)


remove_non_ascii('slabiky, ale liší se podle významu')
> slabiky, ale li se podle vznamu

Titanomachy

Hi, while this solution looks nice (I would like to avoid exceptions whenever it is possible), it does not recognize all english characters. Even Space is not recognized. – Hepatica

Answer 4

I believe this one would have a minimal runtime since it stops once it finds a character which is not a latin letter. It also uses a generator for better memory usage.

import string

def has_only_latin_letters(name):
    char_set = string.ascii_letters
    return all((True if x in char_set else False for x in name))

>>> has_only_latin_letters('_1991_اف_جي2')
False
>>> has_only_latin_letters('bla bla')
True
>>> has_only_latin_letters('blä blä')
False
>>> has_only_latin_letters('저주중앙초등학교')
False
>>> has_only_latin_letters('also a string with numbers and punctuation 1, 2, 4')
True

You can also use a different set of characters:

>>> string.ascii_letters
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'

>>> string.ascii_lowercase
'abcdefghijklmnopqrstuvwxyz'

>>> string.ascii_uppercase
'ABCDEFGHIJKLMNOPQRSTUVWXYZ'

>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

>>> string.digits
'0123456789'

>>> string.digits + string.lowercase
'0123456789abcdefghijklmnopqrstuvwxyz'    

>>> string.printable
'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%& 
\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'

To add latin accented letters, you can refer to this post.

Farron

Answer 5

import re

english_check = re.compile(r'[a-z]')

if english_check.match(w):
    print "english",w
else:
    print "other:",w

Eustazio

What about words like naïve or cliché? – Togs

Contrary to the accepted answer this also works for strings with accents :-) (I tested with ['tele', 'tèle', 'τήλε'] and the results are [True, True, False].) – Hexyl

Answer 6

w.isidentifier()

You can easily see the method in docs:

Return true if the string is a valid identifier according to the language definition, section Identifiers and keywords.

Deadfall

does not work: >>> w = '지난해' >>> w.isidentifier() True – Mezereon