Mediawiki parsing in ANTLR: processing ' tokens

Question

I'm trying to write a grammar to parse Media wiki's wiki syntax, and after this the Creole syntax too (unfortunately an existing Creole grammar doesn't work in Antlr 3).

My issue right now is being able to capture a bold rule when I'm already inside an italic rule, or visa versa. For example

'' this text is bold '''now it's italic''' and just bold again''

I've got a lot of help from this question but I'm stuck. The goal is to produce HTML inside the grammar using actions, or possibly an AST - I'm not sure which is best yet.

Versicular

Answer 1

As an exercise, I created a MediaWiki parser as well and didn't match open- and close-tags for bold and italic, but rather invoked a toggle like this:

grammar MediaWiki;

options {
  output=AST;
  backtrack=true;
  memoize=true;
}

...

// entry point of the parser
parse
  :  atom+ EOF -> ^(ROOT atom+)
  ;

atom
  :  formatToggle
  |  horizontalRule
  |  header
  |  link
  |  list
  |  preFormattedText
  |  table
  |  ...
  |  any
  ;

formatToggle
  :  SQt SQt SQt SQt SQt -> BOLD_ITALIC
  |  SQt SQt SQt         -> BOLD
  |  SQt SQt             -> ITALIC
  ;

...

SQt
  :  '\''
  ; 

And then during the translation of the MediaWiki format (to HTML?), you keep flip some boolean flags when you encounter one of BOLD_ITALIC, BOLD or ITALIC.

I haven't tested my grammar properly yet, so I'm not going to post the entire grammar here.

Good luck!

Wearisome

Thanks - did you get anywhere with the mediawiki grammar? I'm beginning to think even Antlr's LL(*) parser can't handle it's poor token design, and I should just turn it into Creole with some string replacement and build a grammar for that instead. – Versicular

@Chris, yeah I am nearly done with it. But there are so many ambiguities, I turned on global backtracking (and to increase performance, enabled memoizing as well). Due to this, ANTLR needs around 20 seconds to create the lexer and parser and I'll have to increase the heap space to 256 MB. Of course, I don't need so much RAM while parsing actual Wiki-sources, only generating the lexer and parser needs quite a bit of heap space. Best of luck! – Wearisome

Sounds like ANTLR isn't a viable solution for a wiki engine (or any other like Gold parser/Grammatica but they break on invalid markup) – Versicular

@Chris, I agree, a single parser/grammar isn't ideal. But, as I suggested in the post you linked to, I think it can be done using multiple parsers, which you could use ANTLR for (but I haven't done this myself). – Wearisome