Scikits-learn: Use custom vocabulary together with Pipeline

Question

In my scikits-learn Pipeline, I would like to pass a custom vocabulary to CountVectorizer():

text_classifier = Pipeline([
    ('count', CountVectorizer(vocabulary=myvocab)),
    ('tfidf', TfidfTransformer()),
    ('clf', LinearSVC(C=1000))
])

However, as far as I understand when I call

text_classifier.fit(X_train, y_train)

Pipeline uses the fit_transform() method of CountVectorizer(), which ignores myvocab. How could I modify my Pipeline to use myvocab? Thanks!

Tattle

Answer 1

This was a bug in scikit-learn that I fixed five minutes ago. Thanks for spotting it. I suggest you either upgrade to the newest version from Github, or separate the vectorizer from the pipeline as a workaround:

count = CountVectorizer(vocabulary=myvocab)
X_vectorized = count.transform(X_train)

text_classifier = Pipeline([
    ('tfidf', TfidfTransformer()),
    ('clf', LinearSVC(C=1000))
])

text_classifier.fit(X_vectorized, y_train)

UPDATE: since this answer was posted, this fix has been incorporated in several scikit-learn releases.

Johann