Document-term matrix in R - bigram tokenizer not working

Question

I am trying to make 2 document-term matrices for a corpus, one with unigrams and one with bigrams. However, the bigram matrix is currently just identical to the unigram matrix, and I'm not sure why.

The code:

docs<-Corpus(DirSource("data", recursive=TRUE))

# Get the document term matrices
BigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
dtm_unigram <- DocumentTermMatrix(docs, control = list(tokenize="words", 
    removePunctuation = TRUE, 
    stopwords = stopwords("english"), 
    stemming = TRUE))
dtm_bigram <- DocumentTermMatrix(docs, control = list(tokenize = BigramTokenizer,
    removePunctuation = TRUE,
    stopwords = stopwords("english"),
    stemming = TRUE))

inspect(dtm_unigram)
inspect(dtm_bigram)

I also tried using ngram(x, n=2) from the ngram package as the tokenizer, but that doesn't work either. How do I fix the bigram tokenization?

Trichromatic

I am also having this issue so if you found the answer please let me know. – Tubule

Bit late on the reply, sorry - but I got this working by using VCorpus instead of Corpus. – Trichromatic

Answer 1

The tokenizer option doesn't seem to work with Corpus (SimpleCorpus). Using VCorpus instead cleared up the problem.

Trichromatic

Why VCorpus over Corpus? There is another related SO problem here but there doesn't seem to be satisfying explanation. – Coordination