Calculate TF-IDF of documents using HBase as the datasource

Question

I want to calculate the TF (Term Frequency) and the IDF (Inverse Document Frequency) of documents that are stored in HBase.

I also want to save the calculated TF in a HBase table, also save the calculated IDF in another HBase table.

Can you guide me through?

I have looked at BayesTfIdfDriver from Mahout 0.4 but I am not getting a head start.

Eade

Mahout 0.2? That's ancient, from 4 years ago. Use 0.6, or 0.7 which is about to be released. – Barbusse

@SeanOwen It is 0.4 corrected it. Apache pulled out the HBase data store from 0.5. So I am using 0.4. – Eade

I deleted it :) The idea is that there doesn't really need to be direct HBase integration. Use Mahout with Hadoop, and Hadoop with HBase. That's the idea. – Barbusse

@SeanOwen Can you just tell me how can i give input from HBase to Mahout using Hadoop. Please. – Eade

I don't know anything about this part of the code. The question is, how do you use HBase as input for Hadoop? That's a Hadoop question. – Barbusse

Hi Sean. I have a question. I have the tf and the dictionary in HBase. I plan to create tf-idf from these values and pass it to RowSimilarityJob. I want to know, while creating the Sequence file, What does TFIDFPartialVectorReducer do to words which are not in that document. Does it create a "0" value cell or it just ignores it. I plan to create a Sequence file and give it to the RowSimilarityJob. But I do not what to do with the words which are not in the document but are in the dictionary. – Eade

Answer 1

The outline of a solution is pretty straight forward:

1. do a word count over your hbase tables, storing both term frequency and document frequency for each word
2. in your reduce phase aggregate the term frequency and document frequency for each word
3. Given a count of your documents, scan through your aggregated results one more time and calculate the IDF based off of the document frequency.

The wikipedia page on TF-IDF is a good reference to remember the details of the formula: http://en.wikipedia.org/wiki/Tf*idf

Optics

Answer 2

If you want to calculate TF, IDF then you need to create intermediate table "TermMatrix" that stores terms with document IDs. Then you can calculate TFIDF by using the TermMatrix table. It is close to real-time but if you want real-time TFIDF output then I would recommend to create "TF", "IDF" tables too.

I wrote a blog about how to calculate TFIDF by using HBase: http://ahikmat.blogspot.kr/2014/07/building-term-matrix-on-hbase.html

Inboard