How areTF-IDF calculated by the scikit-learn TfidfVectorizer

Question

I run the following code to convert the text matrix to TF-IDF matrix.

text = ['This is a string','This is another string','TFIDF computation calculation','TfIDF is the product of TF and IDF']

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(max_df=1.0, min_df=1, stop_words='english',norm = None)
                    
X = vectorizer.fit_transform(text)
X_vocab = vectorizer.get_feature_names_out()
X_mat = X.todense()
X_idf = vectorizer.idf_

I get the following output

X_vocab =

[u'calculation',
 u'computation',
 u'idf',
 u'product',
 u'string',
 u'tf',
 u'tfidf']

and X_mat =

  ([[ 0.        ,  0.        ,  0.        ,  0.        ,  1.51082562,
      0.        ,  0.        ],
    [ 0.        ,  0.        ,  0.        ,  0.        ,  1.51082562,
      0.        ,  0.        ],
    [ 1.91629073,  1.91629073,  0.        ,  0.        ,  0.        ,
      0.        ,  1.51082562],
    [ 0.        ,  0.        ,  1.91629073,  1.91629073,  0.        ,
      1.91629073,  1.51082562]])

Now I dont understand how these scores are computed. My idea is that for the text[0], score for only 'string' is computed and there is a score in the 5th coloumn. But as TF_IDF is the product of term frequency which is 2 and IDF which is log(4/2) is 1.39 and not 1.51 as shown in the matrix. How is the TF-IDF score calculated in scikit-learn.

Freeborn

Answer 1

TF-IDF is done in multiple steps by Scikit Learn's TfidfVectorizer, which in fact uses TfidfTransformer and inherits CountVectorizer.

Let me summarize the steps it does to make it more straightforward:

1. tfs are calculated by CountVectorizer's fit_transform()
2. idfs are calculated by TfidfTransformer's fit()
3. tfidfs are calculated by TfidfTransformer's transform()

You can check the source code here.

Back to your example. Here is the calculation that is done for the tfidf weight for the 5th term of the vocabulary, 1st document (X_mat[0,4]):

First, the tf for 'string', in the 1st document:

tf = 1

Second, the idf for 'string', with smoothing enabled (default behavior):

df = 2
N = 4
idf = ln(N + 1 / df + 1) + 1 = ln (5 / 3) + 1 = 1.5108256238

And finally, the tfidf weight for (document 0, feature 4):

tfidf(0,4) = tf * idf = 1 * 1.5108256238 = 1.5108256238

I noticed you choose not to normalize the tfidf matrix. Keep in mind normalizing the tfidf matrix is a common and usually recommended approach, since most models will require the feature matrix (or design matrix) to be normalized.

TfidfVectorizer will L-2 normalize the output matrix by default, as a final step of the calculation. Having it normalized means it will have only weights between 0 and 1.

Pearlene

This is really good answer!! I spent whole day understanding this. @Pearlene can you show in this example that how normalization is applied? – Hansom

Great explanation, one quick note here, here the logarithm that sklearn uses is the natural logarithm, if you are deriving this by hand (or calculator) use "ln" instead of log base 10. – Anabal

@Anabal I fixed the portion where I mistakenly used "log" instead of "ln", thank you – Pearlene

Answer 2

The precise computation formula is given in the docs:

The actual formula used for tf-idf is tf * (idf + 1) = tf + tf * idf, instead of tf * idf

and

Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once.

That means 1.51082562 is obtained as 1.51082562=1+ln((4+1)/(2+1))

Bullough

so does 1.51 represent only the IDF score and not the TF-IDF score. TF-IDF score i guess would be 2 *1.51 = 3.02. – Freeborn

The term frequency is just 1, isn't it? That's why we have 1*1.51 – Bullough

Now I go it. Thanks. – Freeborn

Answer 3

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from collections import Counter
corpus = [
     'This is the first document.',
     'This document is the second document.',
     'And this is the third one.',
     'Is this the first document?',
 ]
print(corpus)
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())

z=X.toarray()
#term frequency is printed
print(z)

vectorizer1 = TfidfVectorizer(min_df=1)
X1 = vectorizer1.fit_transform(corpus)
idf = vectorizer1.idf_
print (dict(zip(vectorizer1.get_feature_names(), idf)))
#printing idf
print(X1.toarray())
#printing tfidf

#formula 
# df = 2
# N = 4
# idf = ln(N + 1 / df + 1) + 1 = log (5 / 3) + 1 = 1.5108256238

#formula
# tfidf(0,4) = tf * idf = 1 * 1.5108256238 = 1.5108256238

Lampion