Compute word n-grams on original text or after lemma/stemming process?

Question

I'm thinking about use word n-grams techniques on a raw text. But I have a doubt:

does it have sense use word n-grams after applying lemma/stemming on text? If not, why should I use word n-grams only on raw files? What are pros and cons?

Olivares

Are you asking about character n-grams? (e.g. food -> "<fo", "foo", "ood", "od>") or about word n-grams? (e.g. "stem original text" -> "stem original", "original text".) – Shote

Forgot about it. I'm talking about word n-grams. Thanks. – Olivares

Answer 1

Computing word n-grams after lemmatization or stemming would be done for the same reasons as you would want to before stemming. Sometimes this gets you false positives, e.g., (D3) but it usually increases recall in such a meaningful way that you want to do it.

In some domains, e.g., short-text, stemming can hurt. The best thing to do is to test, but in general, I would suggest stemming and case-folding, but it really depends on your domain and queries.

Q="criminal records"

• D1 = "... has a criminal record ..." (match on stem)
• D2 = "... released the criminal records ..." (match normally)
• D3 = "... while working on 'Smooth Criminal', recording ..." (false match on stem)

It's a precision/recall tradeoff. You can increase recall by stemming (always) and you can increase precision by not stemming. But it depends on what kinds of queries you are serving. If you're running code search, for instance, you almost never want to stem or preprocess, because users expect to type in exact symbol names and then find them.

Shote