Web page recommender system

Question

I am trying to build a recommender system which would recommend webpages to the user based on his actions(google search, clicks, he can also explicitly rate webpages). To get an idea the way google news does it, it displays news articles from the web on a particular topic. In technical terms that is clustering, but my aim is similar. It will be content based recommendation based on user's action.

So my questions are:

1. How can I possibly trawl the internet to find related web-pages?
2. And what algorithm should I use to extract data from web-page is textual analysis and word frequency the only way to do it?
3. Lastly what platform is best suited for this problem. I have heard of Apache mahout and it comes with some re-usable algos, does it sound like a good fit?

Leaves

You can write multiple books about your questions. – Tuchun

haha, do they sound like big problems? Well I am looking for a simple solution. Just a brief description on how to approach such a project. – Leaves

Programming Collective Intelligence is a easy, math-less introduction to Machine Learning and contains a use-case like your question. – Sprague

Answer 1

as Thomas Jungblut said, one could write several books on your questions ;-) I will try to give you a list of brief pointers - but be aware there will be no ready-to-use off-the-shelf solution ...

1. Crawling the internet: There are plenty of toolkits for doing this, like Scrapy for Python , crawler4j and Heritrix for Java, or WWW::Robot for Perl. For extracting the actual content from web pages, have a look at boilerpipe.

   http://scrapy.org/

   http://crawler.archive.org/

   http://code.google.com/p/crawler4j/

   https://metacpan.org/module/WWW::Robot

   http://code.google.com/p/boilerpipe/

2. First of all, often you can use collaborative filtering instead of content-based approaches. But if you want to have good coverage, especially in the long tail, there will be no way around analyzing the text. One thing to look at is topic modelling, e.g. LDA. Several LDA approaches are implemented in Mallet, Apache Mahout, and Vowpal Wabbit. For indexing, search, and text processing, have a look at Lucene. It is an awesome, mature piece of software.

   http://mallet.cs.umass.edu/

   http://mahout.apache.org/

   http://hunch.net/~vw/

   http://lucene.apache.org/

3. Besides Apache Mahout which also contains things like LDA (see above), clustering, and text processing, there are also other toolkits available if you want to focus on collaborative filtering: LensKit, which is also implemented in Java, and MyMediaLite (disclaimer: I am the main author), which is implemented in C#, but also has a Java port.

   http://lenskit.grouplens.org/

   http://ismll.de/mymedialite

   https://github.com/jcnewell/MyMediaLiteJava

Asare

thanks zenog for your descriptive answer. That's the answer which I was looking for, much appreciated :) – Leaves

Answer 2

This should be a good read: Google news personalization: scalable online collaborative filtering

It's focused on collaborative filtering rather than content based recommendations, but it touches some very interesting points like scalability, item churn, algorithms, system setup and evaluation.

Mahout has very good collaborative filtering techniques, which is what you describe as using the behaviour of the users (click, read, etc) and you could introduce some content based using the rescorer classes.

You might also want to have a look at Myrrix, which is in some ways the evolution of the taste (aka recommendations) portion of Mahout. In addition, it also allows applying content based logic on top of collaborative filtering using the rescorer classes.

If you are interested in Mahout, the Mahout in Action book would be the best place to start.

Necrose

Thank you Astatic. I haven't read through the paper you suggested, but hopefully it covers how google finds web pages from all across the web compared to articles from a single site. Myrrix looks interesting need to look more into it. Thanks again – Leaves