Which Perl modules for good for data munging?

Question

Nine years ago when I started to parsing HTML and free text with Perl I read the classic Data Munging with Perl. Does someone know if David is planning to update the book or if there are similar books or web pages where the new parsing modules like XML-Twig, Regexp-Grammars, etc, are explained?

I assume that in the last nine years some modules still are as good as they were, some are up to date but with new interesting methods and some have better replacements. For example, is still Parse-RecDescent the only option for free text parsing or will be the Perl 6 influenced Regexp-Grammars its replacement in many scenarios?

I have been four years without active HTML, XML or free text data mining with Perl, so probably my toolkit in this area is a bit outdated. Therefore any feedback for HTML and DOM manipulation, link extraction/verification, web-testing like Mechanize, XML manipulation and free text parsing , from people that is up to date with the current CPAN modules in this area will be more than welcome.

Some new additions to my toolkit:

• JQuery
• pQuery
• HTML-SimpleLinkExtor
• XML::Twig
• HTML-Table

still in my toolkit:

• HTML-TableExtract # not updated since 2006
• WWW-Mechanize
• Parse-RecDescent
• HTML-TokeParser
• URI-Escape
• [more...]

Hebdomad

Answer 1

It's unlikely that there will ever be a second edition of "Data Munging with Perl". I'm afraid that the economics just don't stack up.

But, you're right that technology has moved on a long way since 2001 and there are plenty of new and improved modules that cover much of the same area as the modules discussed in the book, For example, I can't remember the last time I used XML::Parser or XML::DOM. I seem to use XML::LibXML for the majority of my XML work these days. Also, of course, my discussion of databases is incomplete because it doesn't mention DBIx::Class.

Perhaps it would be an interesting idea to update some of this information through some posts on my Perl blog. I'll give it some thought. Thanks for the idea.

Syllabus

Some of the specifics might have changed, but the concepts are the same. :) – Qualitative

Dave, it would be a pleasure to read these reviews and recipes in your blog some day. – Hebdomad

Yes, I'd be VERY interested in reading those post as well. If you do end up posting some additional updates on your blog, that be super. Cheers! – Catarinacatarrh

Answer 2

re: Parse::RecDescent <=> Regexp::Grammars

Damian Conway has been quoted saying that Regexp::Grammars is the successor to Parse::RecDescent. But even so if Parse::RecDescent still gets the job done for you then continue to use it. The tool you know well is better than the tool you don't know!

However if performance is a key issue and you are running perl 5.10+ then do consider Regexp::Grammars.

Hope Dave doesn't mind but here is his first Parse::RecDescent example from Data Munging with Perl (11.1.1) converted to Regexp::Grammars:

use 5.010;
use warnings;
use Regexp::Grammars;

my $parser = qr{
    <Sentence>

    <rule: Sentence>        <subject> <verb> <object>
    <rule: subject>         <noun_phrase>
    <rule: object>          <noun_phrase>
    <rule: noun_phrase>     <pronoun> | <proper_noun> | <article> <noun>

    <token: verb>           wrote | likes | ate
    <token: article>        a | the | this
    <token: pronoun>        it | he
    <token: proper_noun>    Perl | Dave | Larry
    <token: noun>           book | cat
}xms;

while (<DATA>) {
    chomp;
    print "'$_' is ";
    print 'NOT ' unless $_ =~ $parser;
    say 'a valid sentence';
}

__DATA__
Larry wrote Perl
Larry wrote a book
Dave likes Perl
Dave likes the book
Dave wrote this book
the cat ate the book
Dave got very angry

NB. For those you don't have the book only "Dave got very angry" is an invalid sentence :)

/I3az/

Subtemperate

@Mike: Its annoying you can't copy/paste from printed paper to here :) Well spotted I've s/noun_phase/noun_phrase/ – Subtemperate

Of course "Dave got very angry" isn't a valid sentence. Dave never gets angry :-) – Syllabus

@draegtun: +1 Thanks a lot for the example. I have been without using P::RD for 5 years so I would need to read again the pod. Therefore if the concepts of tokens, production rules etc are similar, then from your answer I assume that is wiser to write my new parsers with R::G, isn't it?. – Hebdomad

@Pablo Marin-Garcia: If i were to start a new parsing project tomorrow then i would choose R::G. I would only use P::RD if the project had to run on a perl prior to 5.10. – Subtemperate

@draegtun: thanks for your suggestion. I am using perl 5.12 so I will give a chance to R::G. I hope that the P::RD tutorials and documentation around could be easily transformed to R::G. – Hebdomad