Error using langdetect in python: "No features in text"

Question

Hey I have a csv with multilingual text. All I want is a column appended with a the language detected. So I coded as below,

from langdetect import detect 
import csv
with open('C:\\Users\\dell\\Downloads\\stdlang.csv') as csvinput:
with open('C:\\Users\\dell\\Downloads\\stdlang.csv') as csvoutput:
writer = csv.writer(csvoutput, lineterminator='\n')
reader = csv.reader(csvinput)

    all = []
    row = next(reader)
    row.append('Lang')
    all.append(row)

    for row in reader:
        row.append(detect(row[0]))
        all.append(row)

    writer.writerows(all)

But I am getting the error as LangDetectException: No features in text

The traceback is as follows

runfile('C:/Users/dell/.spyder2-py3/temp.py', wdir='C:/Users/dell/.spyder2-py3')
Traceback (most recent call last):

  File "<ipython-input-25-5f98f4f8be50>", line 1, in <module>
    runfile('C:/Users/dell/.spyder2-py3/temp.py', wdir='C:/Users/dell/.spyder2-py3')

  File "C:\Users\dell\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "C:\Users\dell\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 89, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/dell/.spyder2-py3/temp.py", line 21, in <module>
    row.append(detect(row[0]))

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector_factory.py", line 130, in detect
    return detector.detect()

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector.py", line 136, in detect
    probabilities = self.get_probabilities()

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector.py", line 143, in get_probabilities
    self._detect_block()

  File "C:\Users\dell\Anaconda3\lib\site-packages\langdetect\detector.py", line 150, in _detect_block
    raise LangDetectException(ErrorCode.CantDetectError, 'No features in text.')

LangDetectException: No features in text.

This is how my csv looks like 1)skunkiest smokiest yummiest strain pain killer and mood lifter 2)Relaxation, euphorique, surélevée, somnolence, concentré, picotement, une augmentation de l’appétit, soulager la douleur Giggly, physique, esprit sédation 3)Reduzierte Angst, Ruhe, gehobener Stimmung, zerebrale Energie, Körper Sedierung 4)Calmante, relajante muscular, Relajación Mental, disminución de náuseas 5)重いフルーティーな幸せ非常に強力な頭石のバースト

Please help me with this.

Mackle

you cannot read and write the same file! use another file for output! – Bollen

Could you include the first few lines of the CSV so we could take a look? – Versicolor

What is the content of row[0] when this error is thrown? – Cicerone

Perhaps you could also add line counting to the loop, so that, when the error happens, it will be possible to know exactly which of the rows caused it. – Versicolor

@Jean-FrançoisFabre tried with different file as output, but that doesn't work! – Mackle

@Versicolor please see to the edited question to have an idea of how my csv looks like. – Mackle

On the Python console, you could try to manually call the detect() function for each of those strings to see which is causing the problem. – Versicolor

Answer 1

You can use something like this to detect which line in your file is throwing the error:

for row in reader:
    try:
        language = detect(row[0])
    except:
        language = "error"
        print("This row throws and error:", row[0])
    row.append(language)
    all.append(row)

What you're going to see is that it probably fails at "重いフルーティーな幸せ非常に強力な頭石のバースト". My guess is that detect() isn't able to 'identify' any characters to analyze in that row, which is what the error implies.

Other things, like when the input is only a URL, also cause this error.

Polity

Isn't the point of this package to detect languages that do not contain letters like Japanese, Chinese, etc? I don't get an error when running detect('重いフルーティーな幸せ非常に強力な頭石のバースト') – Decoupage

this is an old post, so maybe the package has been updated? when i ran it in 2017 that's where it failed. – Polity

Answer 2

The error occurred when passing an object with no letters to detect. At least one letter should be there.

To reproduce, run any of below commands:

detect('.')
detect(' ')
detect('5')
detect('/')

So, you may apply some text pre-processing first to drop records in which row[0] value is an empty string,a null value, a white space, a number, a special character, or simply doesn't include any alphabets.

Fino

This was my case, and one can simply check the string using bool(re.match('^(?=.*[a-zA-Z])', your_string)) with import re to see whether there is at least one alphabet in your string – Biodynamics

Answer 3

the problem is a null text or something like ' ' with no value; check this in a condition and loop your reader in a list comprehension or

from langdetect import detect   
textlang = [detect(elem) for elem in textlist if len(elem) > 50]

textlang = [detect(elem) if len(elem) > 50 else elem == 'no' for elem in textlist]

or with a loop

  texl70 = df5['Titletext']
  langdet = []                                                    

  for i in range(len(df5)):                                         
    try:                                                          
       lang=detect(texl70[i])                                      
    except:                                                       
       lang='no'                                                  
       print("This row throws error:", texl70[i])                 
    langdet.append(lang)                

Score

>>> textlang = [detect(elem) if len(elem) > 50 else elem == 'no' for elem in textlist] Its better to keep your list in sync and add a false if no lang detected – Score

This error is not just for empty strings, but for text containing only an email address, URL, a sequence of numbers, etc. Checking these is a bit tricky, but error handling shown above should be helpful. – Junior

Right, there's less sense to detect lang in an email or URL, Thanks – Score

Answer 4

The error occurrs when string has no letters. If you want to ignore that row and continue the process.

for i in df.index:
  str = df.iloc[i][1]  
  try:
    lang = detect(str)
  except:
    continue 

Agglutinate

Answer 5

It is a bad practice to catch all possible exceptions. Let me propose something more complete, more readable and safer:

rx_letters = re.compile("[a-z]+", re.I)

for row in reader:
    try:
        if rx_letters.search(row[0]) is not None:
            row.append(detect(row[0]))
    except LangDetectException as e:
        row.append("?")
        print(f"Lang detect failed for: '{row[0]}'")

rx_letters check can be skipped, but I find it more elegant to check for the most basic condition.

Peery