Antlr4 - Is there a simple example of using the ParseTree Walker?

Question

Antlr4 has a new class ParseTreeWalker. But how do I use it? I am looking for a minimal working example. My grammar file is 'gram.g4' and I want to parse a file 'program.txt'

Here is my code so far. (This assumes ANTLR has run my grammar file and created all of the gramBaseListener, gramLexer, etc etc):

import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import static org.antlr.v4.runtime.CharStreams.fromFileName;

public class launch{
public static void main(String[] args) {

    CharStream cs = fromFileName("gram.g4");  //load the file
    gramLexer lexer = new gramLexer(cs);  //instantiate a lexer
    CommonTokenStream tokens = new CommonTokenStream(lexer); //scan stream for tokens
    gramParser parser = new gramParser(tokens);  //parse the tokens

    // Now what??  How do I connect the above with the below? 

    ParseTreeWalker walker = new ParseTreeWalker();  // how do I use this to parse program.txt??
}}

I am using java but I assume it is similar in other languages.

The ANTLR documentation (http://www.antlr.org/api/Java/index.html) is short on examples. There are many tutorials on the internet but they are mostly for ANTLR version 3. The few using version 4 don't work or are outdated (for example, there is no parser.init() function, and classes like ANTLRInputStream are depreciated)

Thanks in advance for anyone who can help.

Luca

Hi, I'm sorry to respond to this after such a long time. Could you include your simple grammar file? I think the answers could be more obvious if all of the relevant sources were included. – Scrofula

Answer 1

For each of your parser rules in your grammar the generated parser will have a corresponding method with that name. Calling that method will start parsing at that rule.

Therefore if your "root-rule" is named start then you'd start parsing via gramParser.start() which returns a ParseTree. This tree can then be fed into the ParseTreeWalker alongside with the listener you want to be using.

All in all it could look something like this (EDITED BY OP):

import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import static org.antlr.v4.runtime.CharStreams.fromFileName;

public class launch{
public static void main(String[] args) {

    CharStream cs = fromFileName("program.txt");  //load the file
    gramLexer lexer = new gramLexer(cs);  //instantiate a lexer
    CommonTokenStream tokens = new CommonTokenStream(lexer); //scan stream for tokens
    gramParser parser = new gramParser(tokens);  //parse the tokens

    ParseTree tree = parser.start(); // parse the content and get the tree
    Mylistener listener = new Mylistener();

    ParseTreeWalker walker = new ParseTreeWalker();
    walker.walk(listener,tree);
}}

************ NEW FILE Mylistener.java ************

public class Mylistener extends gramBaseListener {
        @Override public void enterEveryRule(ParserRuleContext ctx) {  //see gramBaseListener for allowed functions
            System.out.println("rule entered: " + ctx.getText());      //code that executes per rule
        }
    }

Of course you have to replace <listener> with your implementation of BaseListener

And just one small sidenode: In Java it is convention to start classnames with capital letters and I'd advise you to stick to that in order for making the code more readable for other people.

Their

Error: java: cannot find symbol symbol: method start() location: variable parser of type gramParser – Luca

AH, ok, start() is the actual name of my rule inside gram.g4. Sorry still new to all this. A few questions. So listener is created by gramBaseListener gbl = new gramBaseListener(); Now how do I have it parse a program? – Luca

Yes exactly.. I just used start() as an example as I don't know the names of your grammar rules. The Baselistener doesn't do anything. You have to extend it and overwrite the respective methods you want to perform some code on. – Their

The file is being pasted at the moment you call start(). And as stated above: In order to see anything from that you have to overwrite some methods in the listener. Another good way is to use the TestRig that is shipped with ANTLR for displaying the parse tree graphically. You can search for that on the net... – Their

Ah, ok, thanks. But everywhere I read says ANTLR creates a listener for you automatically. Are you saying it creates an empty listener?? Because that's kind of stupid. Extend that logic to parse trees, tokens, parse files, etc etc. – Luca

Yea it does create an empty listener. This comes in handy if you want to only perform code on some parser rule blocks and not at all of them. If it didn't create the empty listener you'd have to overwrite every single method instead of only those you actually need... – Their

Your edits seem to be correct. Only thing is that in the listener rule you'd print out the text corresponding to the respective token and not the name as you'd might expect... – Their

Upvote straight away for- "For each of your parser rules in your grammar the generated parser will have a corresponding method with that name. Calling that method will start parsing at that rule." – Doorkeeper

Answer 2

This example should work with ANTLR 4.8.

Below the example you can find references to setup your Java env, API and Listeners.

public class Launch {
    public static void main(String[] args) {
        InputStream inputStream = null;
        MyprogramLexer programLexer = null;
        try {
            File file = new File("/program.txt");
            inputStream = new FileInputStream(file);
            programLexer = new MyprogramLexer(CharStreams.fromStream(inputStream)); // read your program input and create lexer instance
        } finally {
            if (inputStream != null) {
                inputStream.close();
            }
        }
        
        /* assuming a basic grammar:
            myProgramStart: TOKEN1 otherRule TOKEN2 ';' | TOKENX finalRule ';'
            ...
        */
        CommonTokenStream tokens = new CommonTokenStream(programLexer); // get tokens
        MyParser parser = new MyParser(tokens);
        MyProgramListener listener = new MyProgramListener();   // your custom extension from BaseListener
        parser.addParseListener(listener);
        parser.myProgramStart().enterRule(listener);    // myProgramStart is your grammar rule to parse

        // what we had built?
        MyProgram myProgramInstance = listener.getMyProgram();    // in your listener implementation populate a MyProgram instance
        System.out.println(myProgramInstance.toString());
    }
}

References:

• https://www.antlr.org/api/Java/
• https://tomassetti.me/antlr-mega-tutorial/#java-setup
• https://riptutorial.com/antlr/example/16571/listener-events-using-labels

Industrialism

Are there any benefits of using this technique over e.g. ParseTreeWalker? When I use this technique with my subclass of TypeScriptParserBaseListener somehow in all my listener methods ctx.getText() is always an empty string? – Hessler