Enabling soundex/metaphone for non-English characters

Question

I've been studying soundex, metaphone and other string search techniques the past few days, and in my understanding both algorithms work well in handling non-English words transliterated to English.

However the requirement that I have would be for such search to work in the original, untransliterated languages, accomodating alphabets such as German, Norwegian, and even Cyrilic alphabets.

Are there any search algorithms capable of handling these alphabets completely? Or am I better off using third party full-text-search libraries such as Lucene? Consequently, the question then becomes 'does Lucene handle non-English alphabets?'

Cloyd

If your use-case is just textual search in non-English languages, you may not need soundex. You do need Lucene with a proper Analyzer, as ire_and_curses said. If you want to handle different writing variants of the same word, you will need a phonetic matching algorithm. Can you say more about your use-case? – Beale

Answer 1

I'm not an expert in this area, but your requirements seem quite difficult to me. Soundex was specifically designed for English sounds as well as characters. I don't think it will perform well for non-English languages. See for example the responses to this related question.

Double-Metaphone is an attempt to deal with much more complex variations than Soundex or Metaphone, and was designed to handle irregularities in a range of languages. It might be sufficient for your needs. There is a list of library implementations on the linked page.

Support for other languages in Lucene is based on the concept of Analyzers. Lucene comes with a set of analyzers for different languages (although I couldn't find the default list), but the quality may be quite variable.

Mclyman

Looks like Lucene + analyzers is what I'm really looking for, thanks. :) – Cloyd

what about Chinese,Japanese,Arabic and Indian languages? – Springer

Answer 2

There are some good references on Wikipedia, starting from the Soundex article. I don't know whether there are existing libraries designed to handle such a wide variety of languages.

Goodoh

The references there all point towards algorithms that handle Anglicized spellings of european names. I haven't seen any that actually handles the special characters as-is -- unless I understand them wrongly. – Cloyd

I thought that maybe some of the Soundex variants were designed to work better with other languages, but they are kind of focused on English or anglicized spellings. It wouldn't be hard to write something LIKE Soundex for each of those languages, but you'd probably need the help of a linguist, if you're not a native speaker. – Goodoh