Processing a String with ANTLR4

Question

I'm trying to convert my grammar from v3 to v4 and having some trouble finding all the right pieces.

In v3 to process a String, I used:

public static DataExtractor create(String dataspec) {
    CharStream stream = new ANTLRStringStream(dataspec);
    DataSpecificationLexer lexer = new DataSpecificationLexer(stream);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    DataSpecificationParser parser = new DataSpecificationParser(tokens);

    return parser.dataspec();
}

How do I change this to work in v4?

Salivate

Answer 1

The changes that were made are:

• ANTLRStringStream has been replaced with a constructor in ANTLRInputStream that takes a String
• parser rules now return a context object which has a public field named according to the returns clause of your rule.

So if the dataspec rule says "returns [DataExtractor extractor]", v4 the method becomes:

public static DataExtractor create(String dataspec) {
    CharStream stream = new ANTLRInputStream(dataspec);
    DataSpecificationLexer lexer = new DataSpecificationLexer(stream);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    DataSpecificationParser parser = new DataSpecificationParser(tokens);

    return parser.dataspec().extractor;
}

Salivate

Answer 2

For ANTLR 4.7 the API was changed a little (ANTLRInputStream is deprecated):

InputStream stream = new ByteArrayInputStream(input.getBytes(StandardCharsets.UTF_8));
lexer.setInputStream(CharStreams.fromStream(stream, StandardCharsets.UTF_8));
parser.setInputStream(new CommonTokenStream(lexer));

Hint: if you want to re-use the parser+lexer instances, call their 'reset()' methods after setting their input streams.

Farro

There is also a CharStreams#fromString in the class. This helps even more if one only got a String and wants it to be lexed and parsed. But thank you for the hint :) – Leal