Computing TF-IDF on the whole dataset or only on training data?

Question

In the chapter seven of this book "TensorFlow Machine Learning Cookbook" the author in pre-processing data uses fit_transform function of scikit-learn to get the tfidf features of text for training. The author gives all text data to the function before separating it into train and test. Is it a true action or we must separate data first and then perform fit_transform on train and transform on test?

Neomaneomah

When in such scenarios, always think what should be done in real -world problem. There you treat all available data as train and new unseen data as test. Now since the test data will not be available in such a case, what you would do. The process of splitting the available data into train and test is to replicate the same. – Salo

@Vivek Kumar Yes, in fact you mean that the approach of fit_transform on all data is not true because we have not test data in real problems. – Neomaneomah

Yes. So only do fit() or fit_transform() on train data. – Salo

Answer 1

According to the documentation of scikit-learn, fit() is used in order to

Learn vocabulary and idf from training set.

On the other hand, fit_transform() is used in order to

Learn vocabulary and idf, return term-document matrix.

while transform()

Transforms documents to document-term matrix.

On the training set you need to apply both fit() and transform() (or just fit_transform() that essentially joins both operations) however, on the testing set you only need to transform() the testing instances (i.e. the documents).

Remember that training sets are used for learning purposes (learning is achieved through fit()) while testing set is used in order to evaluate whether the trained model can generalise well to new unseen data points.

---

For more details you can refer to the article fit() vs transform() vs fit_transform()

Bailor

Yes, that is true, we should not use test data in training. If we fit_transform on all data set, it means we are using test data at training level. also the resulting vocabulary in this two ways are different. – Neomaneomah

Answer 2

Author gives all text data before separating train and test to function. Is it a true action or we must separate data first then perform tfidf fit_transform on train and transform on test?

I would consider this as already leaking some information about the test set into the training set.

I tend to always follow the rule that before any pre-processing first thing to do is to separate the data, create a hold-out set.

Stelmach

Answer 3

As we are talking about text data, we have to make sure that the model is trained only on the vocabulary of the training set as when we will deploy a model in real life, it will encounter words that it has never seen before so we have to do the validation on the test set keeping that in mind.
We have to make sure that the new words in the test set are not a part of the vocabulary of the model.
Hence we have to use fit_transform on the training data and transform on the test data. If you think about doing cross validation, then you can use this logic across all the folds.

Ashelman