Java Lucene NGramTokenizer

Question

I am trying tokenize strings into ngrams. Strangely in the documentation for the NGramTokenizer I do not see a method that will return the individual ngrams that were tokenized. In fact I only see two methods in the NGramTokenizer class that return String Objects.

Here is the code that I have:

Reader reader = new StringReader("This is a test string");
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);

1. Where are the ngrams that were tokenized?
2. How can I get the output in Strings/Words?

I want my output to be like: This, is, a, test, string, This is, is a, a test, test string, This is a, is a test, a test string.

Deppy

Answer 1

I don't think you'll find what you're looking for trying to find methods returning String. You'll need to deal with Attributes.

Should work something like:

Reader reader = new StringReader("This is a test string");
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);
CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);
gramTokenizer.reset();

while (gramTokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    //Do something
}
gramTokenizer.end();
gramTokenizer.close();

Be sure to reset() the Tokenizer it if it needs to be reused after that, though.

---

Tokenizing grouping of words, rather than chars, per comments:

Reader reader = new StringReader("This is a test string");
TokenStream tokenizer = new StandardTokenizer(Version.LUCENE_36, reader);
tokenizer = new ShingleFilter(tokenizer, 1, 3);
CharTermAttribute charTermAttribute = tokenizer.addAttribute(CharTermAttribute.class);

while (tokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    //Do something
}

Aloisia

What can I do with Strings instead of chars in terms of Attributes? So then my output would be something like: This, is, a, test, string, This is, is a, a test, ... a test string. – Deppy

Okay, that's not what Lucene's NGramTokenizer is designed to handle. What you'll want to use, I think, is a ShingleFilter combined with StandardTokenizer. I'll update my answer, easier to express there... – Aloisia

Do you know of any stop word filters I can use in the tokenization process? – Deppy

I think the standard would be StopFilter. Another very typical filter to apply would be StandardFilter, by the way. I would think StandardFilter, then StopFilter, then ShingleFilter would probably get good results. – Aloisia

Took a look at my latest post involving the StopFilter : #13501921 – Deppy

Where did tokenStream come from? – Vela

@Vela It must be a typo. It should have been tokenizer. – Froe

what is your maven dependency of lucene NGramTokenizer? – Galegalea

Is there a way I can preserve the spaces as is in the input? For example, for the input abc xyz mno, I want abc xyz instead of abc xyz – Semipermeable

Answer 2

For recent version of Lucene (4.2.1), this is a clean code which works. Before executing this code, you have to import 2 jar files:

• lucene-core-4.2.1.jar
• lucene-analuzers-common-4.2.1.jar

Find these files at http://www.apache.org/dyn/closer.cgi/lucene/java/4.2.1

//LUCENE 4.2.1
Reader reader = new StringReader("This is a test string");      
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);

CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);

while (gramTokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    System.out.println(token);
}

Podagra

Answer 3

Without creating a test program, I would guess that incrementToken() returns the next token which will be one of the ngrams.

For example, using ngram lengths of 1-3 with the string 'a b c d', NGramTokenizer could return:

a
a b
a b c
b
b c
b c d
c
c d
d

where 'a', 'a b', etc. are the resulting ngrams.

[Edit]

You might also want to look at Querying lucene tokens without indexing, as it talks about peeking into the token stream.

Flume

Answer 4

package ngramalgoimpl;
import java.util.*;

public class ngr {

    public static List<String> n_grams(int n, String str) {
        List<String> n_grams = new ArrayList<String>();
        String[] words = str.split(" ");
        for (int i = 0; i < words.length - n + 1; i++)
            n_grams.add(concatination(words, i, i+n));
        return n_grams;
    }
     /*stringBuilder is used to cancatinate mutable sequence of characters*/
    public static String concatination(String[] words, int start, int end) {
        StringBuilder sb = new StringBuilder();
        for (int i = start; i < end; i++)
            sb.append((i > start ? " " : "") + words[i]);
        return sb.toString();
    }

    public static void main(String[] args) {
        for (int n = 1; n <= 3; n++) {
            for (String ngram : n_grams(n, "This is my car."))
                System.out.println(ngram);
            System.out.println();
        }
    }
}

Sovran

please provide context, what does this code do and how does it provide an answer on the question? – Fusil

@KevinKloet see the question and given answer – Sovran