How to extract Article Text contents from HTML page like Pocket (Read It Later) or Readability? [closed]

Question

I am looking for some open source framework or algorithm to extract article text contents from any HTML page by cleaning the HTML code, removing garbage stuff, similar to what Pocket (aka Read It Later) software does.

Pocket official webpage: http://getpocket.com/

This question is already available under link: How to extract text contents from html like Read it later or InstaPaper Iphone app? but my requirement is bit different. I want to clean the HTML and extract main contents with images by preserving the font and style (CSS).

Fogel

Answer 1

I would recommend NReadability, together with HtmlAgilityPack

Main text is always in div with id readInner after NReadability transcoded the page.

//** replace this with any url **
string url = "http://www.bbc.co.uk/news/world-asia-19457334";

var t = new NReadability.NReadabilityWebTranscoder();
bool b;
string page = t.Transcode(url, out b);

if (b)
{
    HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    doc.LoadHtml(page);

    var title = doc.DocumentNode.SelectSingleNode("//title").InnerText;
    var imgUrl = doc.DocumentNode.SelectSingleNode("//meta[@property='og:image']").Attributes["content"].Value;
    var mainText = doc.DocumentNode.SelectSingleNode("//div[@id='readInner']").InnerText;
}

Louth

sorry, but why use NReadability if you use HtmlAgilityPack? – Figured

#4183094 – Figured

The "documentation" for nReadability seems to suggest it is simply a pretty printer for HTML (a function that the HAP also has). – Epicontinental

@Oded, Would you test the above code before commenting please. I used it before and know what it does. It really does what OP wants ("cleaning the html code") – Louth

No offence meant - there is a lack of documentation about the library and all I can see in the site suggests it is a pretty printer. I also seem to miss where the OP asks for "cleaning the html code" in the question. – Epicontinental

This usage of Transcode is obsolote (a warning is issued), and the following should be used instead: var transcoder = new NReadabilityWebTranscoder(); transcoder.Transcode(new WebTranscodingInput(yourUrl)); – Carlettacarley

Answer 2

Use the HTML Agilty Pack - it is an open source HTML parser for .NET.

What is exactly the Html Agility Pack (HAP)?

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

You can use this to query HTML and extract whatever data you wish.

Epicontinental

HAP is cool but I prefer ScrapySharp - build over HTML Agility Pack. It adds support of CSS selectors to HAP. nuget.org/packages/ScrapySharp – Illsorted

I tired to use it already but i couldn't obtain desired results with it. Can you guide me how to extract article kind of stuff (i.e. text including sample code with code style and images). – Fogel

@Illsorted - Nice one! Didn't know about it and will probably start using it when I next need to scrape HTML. – Epicontinental

@furqan.safdar - "couldn't obtain desired results" is not very descriptive. You need to give a better definition than that. – Epicontinental

Actually what i am trying to do is i want to generate an article (PDF document using iText and iTextXmlWorker) by cleaning the html code. I just need to capture main text contents, images, and sample code (if any) with original style CSS. I hope it is clear now. – Fogel

@furqan.safdar - That was already clear. What isn't clear is what problems you were having with that. – Epicontinental

@Epicontinental - I might not be familiar with its proper usage for my requirement. Can you please suggest some sample code so i can verify it. – Fogel

@furqan.safdar - The source download of the HAP comes with a bunch of sample projects that show proper usage. – Epicontinental

@Epicontinental - OK i will try it again later. – Fogel