ANTLR 4 and AST visitors

Question

I'm trying to use ASTs with ANTLR4, with this files:

Builder.java

import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CharStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.TokenStream;

public class Builder
{

    public static void main(String[] args)
    {
        CharStream input = new ANTLRInputStream("ON M1==2 && M3 == 5 && (M2 > 1 || M5 <= 5.0) "
                                              + "DO P5:42 P4:10");
        ExprLexer lexer = new ExprLexer(input);
        TokenStream tokens = new CommonTokenStream(lexer);
        ExprParser parser = new ExprParser(tokens);
        parser.addParseListener(new ExprTestListener());
        ExprParser.ExpressionContext uu = parser.expression();
    }

}

ExprTestListener:

import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.Token;
import org.antlr.v4.runtime.tree.TerminalNode;
import org.antlr.v4.runtime.tree.ErrorNode;

public class ExprTestListener extends ExprBaseListener {
    @Override public void enterExpression(ExprParser.ExpressionContext ctx)
    {
        System.out.println(ctx);
    }
    @Override public void exitExpression(ExprParser.ExpressionContext ctx)
    {
        System.out.println(ctx);
    }

    @Override public void enterActionexpr(ExprParser.ActionexprContext ctx)
    {
        System.out.println(ctx);
    }
    @Override public void exitActionexpr(ExprParser.ActionexprContext ctx)
    {
        System.out.println(ctx);
    }

    @Override public void enterCondexpr(ExprParser.CondexprContext ctx)
    {
        System.out.println(ctx);
    }
    @Override public void exitCondexpr(ExprParser.CondexprContext ctx)
    {
        System.out.println(ctx);
    }

    @Override public void enterCond(ExprParser.CondContext ctx)
    {
        System.out.println(ctx);
    }
    @Override public void exitCond(ExprParser.CondContext ctx)
    {
        System.out.println(ctx);
    }

    @Override public void enterEveryRule(ParserRuleContext ctx)
    {
        System.out.println(ctx);
    }
    @Override public void exitEveryRule(ParserRuleContext ctx)
    {
        System.out.println(ctx);
    }
    @Override public void visitTerminal(TerminalNode node)
    {
    }
    @Override public void visitErrorNode(ErrorNode node)
    {
    }
}

Expr.g:

grammar Expr;
options
{
  // antlr will generate java lexer and parser
  language = Java;

}
WS      : [ \t\r\n]+ -> skip ;
OP      : '&&' | '||';
COMP    : '==' | '<' | '>' | '<=' | '>=' | '!=';
fragment INT     : [0-9]+;
REAL    : INT '.' INT | INT;

ACTION  : 'P' INT ':' INT;
MEASURE : 'M' INT;

// ***************** parser rules:
cond       : MEASURE COMP REAL;
condexpr   : '(' condexpr ')' | cond OP condexpr | cond;
actionexpr : ACTION actionexpr | ACTION;
expression : 'ON' condexpr 'DO' actionexpr;

I have this output:

[]
[]
[29]
[29]
[16 29]
[16 29]
[16 29]
[16 29]
[18 29]
[18 29]
[16 18 29]
[16 18 29]
[16 18 29]
[16 18 29]
[18 18 29]
[18 18 29]
[13 18 18 29]
[13 18 18 29]
[16 13 18 18 29]
[16 13 18 18 29]
[16 13 18 18 29]
[16 13 18 18 29]
[18 13 18 18 29]
[18 13 18 18 29]
[20 18 13 18 18 29]
[20 18 13 18 18 29]
[20 18 13 18 18 29]
[20 18 13 18 18 29]
[18 13 18 18 29]
[18 13 18 18 29]
[13 18 18 29]
[13 18 18 29]
[18 18 29]
[18 18 29]
[18 29]
[18 29]
[29]
[29]
[31]
[31]
[24 31]
[24 31]
[24 31]
[24 31]
[31]
[31]
[]
[]

I find it hard to understand visitors with ANTLR4.

I have tree goals:

• Get collect INTs of MEASUREs and ACTIONs (in two different sets)
• Replace some OPs (for example != by <>)
• Get the condexpr (the top item) string with OPs remplaced (se my previous point)

Grudging

get it? 'tree' goals? :) – Eustasius

Answer 1

A tree label you could use, to set the context of the parse and then walks the leaves of the observed graph with a visitor class and trigger methods in order to create actions from the expressions in the language source code. Thus, at the initial visitation, the listener does not process the actual visitor pattern. The actual visitor pattern and the processing through visitation is done by the methods of the expressionbase listener class extension.

The listener identifies the expression:

@Override public void enterListener(ExprParser.EXPR_CONTEXTContext ctx) { 
  //some code to view the compilation process
}

The expression rule gets a name label:

'EXPR_CONTEXT' expression             # EXPR_CONTEXT //the tree label

The expression walker is implemented:

public class ExprWalker extends ExprBaseListener {

  @Override 
  public void enterListener(ExprParser.EXPR_CONTEXTContext ctx) { 

    java.util.List<ExprParser.ExpressionContext> e = ctx.expression();

    System.out.println("EXPRESSION: " //print action
    + e.get(0).getText() + ", " //first element
    + e.get(1).getText() //second element
    + ", " + ... + ", " //number of elements
    + e.get(N).getText()); //last element

}

The main file then walks with the walker:

ParseTree tree = parser.parse(); //parse the tree

intermezzo: before applying the walker visitation pattern one may imagine tree segment optimisation- or processing patterns. The parsed tree could be handled here as a seperate induction of the source code tree. This approach enables more complicated code- and tree processing patterns.

ParseTreeWalker walker = new ParseTreeWalker(); //get the walker
walker.walk(new ExprWalker(), tree); //start visiting

Hothead

There is no need to create a new ParseTreeWalker(). Just use ParseTreeWalker.DEFAULT.walk(...). – Bosomy

Answer 2

First I'll explain what you have observed above:

First and foremost, please read the documentation for the methods you call. The Parser.addParseListener documentation includes the following note:

THIS IS ONLY FOR ADVANCED USERS. Please give your ParseTreeListener to a ParseTreeWalker instead of giving it to the parser!!!!

The implementation of toString() for the ParserRuleContext class simply prints the rule invocation stack at the time the context was created. You are printing this once when the listener enters a rule, and once when it exits. For actionexpr, cond, and condexpr you print it again, resulting in a total of 4 identical output lines for each of those contexts.

Now some notes on your goals:

• Inside of enterCond and exitCond, the MEASURE text is available by calling ctx.MEASURE().getText().
• Inside of enterActionexpr and exitActionexpr, the ACTION text is available by calling ctx.ACTION().getText().
• You can change the COND token by creating a new TerminalNodeImpl and CommonToken for the updated token, and assigning it to the correct index in the field CondContext.children using either a visitor or a listener.

Bosomy