ANTLR4: Unrecognized constant value in a lexer command

Question

I am learning how to use the "more" lexer command. I typed in the lexer grammar shown in the ANTLR book, page 281:

lexer grammar Lexer_To_Test_More_Command ;

LQUOTE : '"'        -> more, mode(STR) ;

WS   : [ \t\r\n]+   -> skip ; 

mode STR ;

STRING : '"'    -> mode(DEFAULT_MODE) ;

TEXT : .        -> more ;

Then I created this simple parser to use the lexer:

grammar Parser_To_Test_More_Command ;

import Lexer_To_Test_More_Command ;

test: STRING EOF ;

Then I opened a DOS window and entered this command:

antlr4 Parser_To_Test_More_Command.g4

That generated this warning message:

warning(155): Parser_To_Test_More_Command.g4:3:29: rule LQUOTE contains a lexer command with an unrecognized constant value; lexer interpreters may produce incorrect output

Am I doing something wrong in the lexer or parser?

Dixson

Answer 1

Combined grammars (which are grammars that start with just grammar, instead of parser grammar or lexer grammar) cannot use lexer modes. Instead of using the import feature¹, you should use the tokenVocab feature like this:

Lexer_To_Test_More_Command.g4:

lexer grammar Lexer_To_Test_More_Command;

// lexer rules and modes here

Parser_To_Test_More_Command.g4:

parser grammar Parser_To_Test_More_Command;

options {
  tokenVocab = Lexer_To_Test_More_Command;
}

// parser rules here

¹ I actually recommend avoiding the import statement altogether in ANTLR. The method I described above is almost always preferable.

Vicegerent

Okay, I made the changes you suggested Sam. I still get this error: Parser_To_Test_More_CommandLexer.java:63: error: cannot find symbol case 0: _mode = STR; break; symbol: variable STR – Dixson

@RogerCostello The file Parser_To_Test_More_CommandLexer.java is left over from when you had a combined grammar. The new source files have slightly different names (specifically, they don't include the Lexer or Parser suffix). You need to clean and rebuild your project. – Vicegerent

Thanks again Sam. I cleaned and rebuilt. That solved the symbol problem but led to a different problem. I ran this command from the DOS prompt: grun Parser_To_Test_More_Command test -gui and that generated this error: Exception in thread "main" java.lang.ClassCastException: class Parser_To_Test_More_Command at java.lang.Class.asSubclass(Unknown Source) at org.antlr.v4.runtime.misc.TestRig.process(TestRig.java:159) at org.antlr.v4.runtime.misc.TestRig.main(TestRig.java:143) QUESTION: Does grun only work with combined grammars? – Dixson

This is exactly what I come across each time I come back to antlr - revert to import statement -> spurious warnings, change to tokenVocab -> ClassCastException. Seems like there should be more documented about what is going on. – Melitta