How to Parse Some Wiki Markup

Question

Hey guys, given a data set in plain text such as the following:

==Events==
* [[312]] – [[Constantine the Great]] is said to have received his famous [[Battle of Milvian Bridge#Vision of Constantine|Vision of the Cross]].
* [[710]] – [[Saracen]] invasion of [[Sardinia]].
* [[939]] – [[Edmund I of England|Edmund I]] succeeds [[Athelstan of England|Athelstan]] as [[King of England]].
*[[1275]] – Traditional founding of the city of [[Amsterdam]].
*[[1524]] – [[Italian Wars]]: The French troops lay siege to [[Pavia]].
*[[1553]] – Condemned as a [[Heresy|heretic]], [[Michael Servetus]] is [[burned at the stake]] just outside [[Geneva]].
*[[1644]] – [[Second Battle of Newbury]] in the [[English Civil War]].
*[[1682]] – [[Philadelphia]], [[Pennsylvania]] is founded.

I would like to end up with an NSDictionary or other form of collection so that I can have the year (The Number on the left) mapping to the excerpt (The text on the right). So this is what the 'template' is like:

*[[YEAR]] – THE_TEXT

Though I would like the excerpt to be plain text, that is, no wiki markup so no [[ sets. Actually, this could prove difficult with alias links such as [[Edmund I of England|Edmund I]].

I am not all that experienced with regular expressions so I have a few questions. Should I first try to 'beautify' the data? For example, removing the first line which will always be ==Events==, and removing the [[ and ]] occurrences?

Or perhaps a better solution: Should I do this in passes? So for example, the first pass I can separate each line into * [[710]] and [[Saracen]] invasion of [[Sardinia]]. and store them into different NSArrays.

Then go through the first NSArray of years and only get the text within the [[]] (I say text and not number because it can be 530 BC), so * [[710]] becomes 710.

And then for the excerpt NSArray, go through and if an [[some_article|alias]] is found, make it only be [[alias]] somehow, and then remove all of the [[ and ]] sets?

Is this possible? Should I use regular expressions? Are there any ideas you can come up with for regular expressions that might help?

Thanks! I really appreciate it.

EDIT: Sorry for the confusion, but I only want to parse the above data. Assume that that's the only type of markup that I will encounter. I'm not necessarily looking forward to parsing wiki markup in general, unless there is already a pre-existing library which does this. Thanks again!

Apotheosis

Answer 1

This code assumes you are using RegexKitLite:

NSString *data = @"* [[312]] – [[Constantine the Great]] is said to have received his famous [[Battle of Milvian Bridge#Vision of Constantine|Vision of the Cross]].\n\
    * [[710]] – [[Saracen]] invasion of [[Sardinia]].\n\
    * [[939]] – [[Edmund I of England|Edmund I]] succeeds [[Athelstan of England|Athelstan]] as [[King of England]].\n\
    *[[1275]] – Traditional founding of the city of [[Amsterdam]].";

    NSString *captureRegex = @"(?i)(?:\\* *\\[\\[)([0-9]*)(?:\\]\\] \\– )(.*)"; 

    NSRange captureRange;
    NSRange stringRange;
    stringRange.location = 0;
    stringRange.length = data.length;

    do 
    {
        captureRange = [data rangeOfRegex:captureRegex inRange:stringRange];
        if ( captureRange.location != NSNotFound )
        {
            NSString *year = [data stringByMatching:captureRegex options:RKLNoOptions inRange:stringRange capture:1 error:NULL];
            NSString *textStuff = [data stringByMatching:captureRegex options:RKLNoOptions inRange:stringRange capture:2 error:NULL];
            stringRange.location = captureRange.location + captureRange.length;
            stringRange.length = data.length - stringRange.location;
            NSLog(@"Year:%@, Stuff:%@", year, textStuff);
        }
    }
    while ( captureRange.location != NSNotFound );

Note that you really need to study up on RegEx's to build these well, but here's what the one I have is saying:

(?i)

Ignore case, I could have left that out since I'm not matching letters.

(?:\* *\[\[)

?: means don't capture this block, I escape * to match it, then there are zero or more spaces (" *") then I escape out two brackets (since brackets are also special characters in a regex).

([0-9]*)

Grab anything that is a number.

(?:\]\] \– )

Here's where we ignore stuff again, basically matching " – ". Note any "\" in the regex, I have to add another one to in the Objective-C string above since "\" is a special character in a string... and yes that means matching a regex escaped single "\" ends up as "\\" in an Obj-C string.

(.*)

Just grab anything else, by default the RegEX engine will stop matching at the end of a line which is why it doesn't just match everything else. You'll have to add code to strip out the [[LINK]] stuff from the text.

The NSRange variables are used to keep matching through the file without re-matching original matches. So to speak.

Don't forget after you add the RegExKitLite class files, you also need to add the special linker flag or you'll get lots of link errors (the RegexKitLite site has installation instructions).

Burrow

Answer 2

I'm no good with regular expressions, but this sounds like a job for them. I imagine a regex would sort this out for you quite easily.

Have a look at the RegexKitLite library.

Lasonyalasorella

Thanks, yeah someone else referred me to that library. (I didn't downvote you). – Akerley

Answer 3

If you want to be able to parse Wikitext in general, you have a lot of work to do. Just one complicating factor is templates. How much effort do you want to go to cope with these?

If you're serious about this, you probably should be looking for an existing library which parses Wikitext. A brief look round finds this CPAN library, but I have not used it, so I can't cite it as a personal recommendation.

Alternatively, you might want to take a simpler approach and decide which particular parts of Wikitext you're going to cope with. This might be, for example, links and headings, but not lists. Then you have to focus on each of these and turn the Wikitext into whatever you want that to look like. Yes, regular expressions will help a lot with this bit, so read up on them, and if you have specific problems, come back and ask.

Good luck!

Servetnick

Thanks I appreciate the response but I don't want to parse the entire wikipedia markup, just that bit that I posted, and it will usually only contain that type of markup. Assume that's the only type of markup that will appear. Sorry for the confusion! – Akerley