Online clustering of news articles

Question

Is there a common online algorithm to classify news dynamically? I have a huge data set of news classified by topics. I consider each of that topics a cluster. Now I need to classify breaking news. Probably, I will need to generate new topics, or new clusters, dynamically.

The algorithm I'm using is the following:

1) I go through a group of feeds from news sites and I recognize news links.

2) For each new link, I extract the content using dragnet, and then I tokenize it.

3) I find the vector representation of all the old news and the last one using TfidfVectorizer from sklearn.

4) I find the nearest neighbor in my dataset computing euclidean distance from the last news vector representation and all the vector representations of the old news.

5) If that distance is smaller than a threshold, I put it in the cluster that the neighbor belongs. Otherwise, I create a new cluster, with the breaking news.

Each time a news arrive, I re-fit all the data using a TfidfVectorizer, because new dimensions can be founded. I can't wait to re-fit once per day, because I need to detect breaking events, which can be related to unknown topics. Is there a common approach more efficient than the one I am using?

Samons

It does not even work reliably off-line, and you want an online algorithm already? – Arsenide

yes, the algorithm I'm using works off-line. tfidf vectorization with knn clusterization is a common approach and it is well know that works fine. I don't understand why you are giving me a bad vote to my question, I'm researching another topic, online clustering, and I need some ideas. – Samons

There is no "knn clustering". Only kNN classification. – Arsenide

ok thanks for your comments! – Samons

I had downvoted because the question wasn't self contained, and thus likely not useful for future visitors. You have improved the question now, so I un-downvoted. Nevertheless, the "clustering" you do is still unreproducible. The second document will have the first as nearest neighbor, and everything is the same "cluster" (whatever a cluster is here anyway) – Arsenide

ok I'm going to explain it better – Samons

Answer 1

If you build the vectorization yourself, adding new data will be much easier.

1. You can trivially add new words as new columns that are simply 0 for all earlier documents.
2. Don't apply the idf weights, but use them as dynamic weights only.

There are well known, and very fast, implementations of this.

For example Apache Lucene. It can add new documents online, and it uses a variant of tfidf for search.

Arsenide

Ok, that apporach could be very useful! – Samons