Is "Implicit token definition in parser rule" something to worry about?

Question

I'm creating my first grammar with ANTLR and ANTLRWorks 2. I have mostly finished the grammar itself (it recognizes the code written in the described language and builds correct parse trees), but I haven't started anything beyond that.

What worries me is that every first occurrence of a token in a parser rule is underlined with a yellow squiggle saying "Implicit token definition in parser rule".

For example, in this rule, the 'var' has that squiggle:

variableDeclaration: 'var' IDENTIFIER ('=' expression)?;

How it looks exactly:

The odd thing is that ANTLR itself doesn't seem to mind these rules (when doing test rig test, I can't see any of these warning in the parser generator output, just something about incorrect Java version being installed on my machine), so it's just ANTLRWorks complaining.

Is it something to worry about or should I ignore these warnings? Should I declare all the tokens explicitly in lexer rules? Most exaples in the official bible The Defintive ANTLR Reference seem to be done exactly the way I write the code.

Clinician

The answer below (https://mcmap.net/q/398360/-is-quot-implicit-token-definition-in-parser-rule-quot-something-to-worry-about) is correct for the question at hand. In case you already did what's recommended there and still get this warning and that's what brought you here (as in my case) you should first try to regenerate all your .tokens-files. It took me way too long to figure that out. – Shakedown

Answer 1

I highly recommend correcting all instances of this warning in code of any importance.

This warning was created (by me actually) to alert you to situations like the following:

shiftExpr : ID (('<<' | '>>') ID)?;

Since ANTLR 4 encourages action code be written in separate files in the target language instead of embedding them directly in the grammar, it's important to be able to distinguish between << and >>. If tokens were not explicitly created for these operators, they will be assigned arbitrary types and no named constants will be available for referencing them.

This warning also helps avoid the following problematic situations:

• A parser rule contains a misspelled token reference. Without the warning, this could lead to silent creation of an additional token that may never be matched.

• A parser rule contains an unintentional token reference, such as the following:

  number : zero | INTEGER;
  zero   : '0'; // <-- this implicit definition causes 0 to get its own token
  

Sergiosergipe

Ok, thank you, I will consider doing this. As for the action code separation, is that something beyond "combined grammar -> AST -> tree gramar with action code"? I only have the book for v3, so if it is something new, is there any article online explaining it? The online documentation on antlr.org seems to lack any information about differences between v3 and v4. – Renegado

ANTLR 4 no longer includes output=AST or tree grammars. Instead, it uses parse trees with listener and/or visitor interfaces which are automatically generated from the parser grammar. – Sergiosergipe

I noticed that ANTLRWorks2 also throws this same message when I accidentally use a token marked as a fragment in a parser rule. Is this intended, or should it show a different error? – Wurtz

@JonSenchyna Often, token types have a 1:1 correspondence with lexer rules. However, token types are not defined for lexer rules marked as fragment, so the warning is still correct - the token was defined solely due to the reference to something with that name in a parser rule. – Sergiosergipe

Given that it's not an error, how do I turn it off? One of the benefits of Antlr is rapid prototyping that allows you to create a nicely structured NL without having to spend time managing a token list up-front. This warning breaks that because now, e.g. in AntlrWorks, I have to keep looking at bogus warnings to go "oh, it's just that stupid warning that I don't care about". – Frisky

@Frisky ANTLR 4 does not have the ability to selectively enable or disable warnings at this time. – Sergiosergipe

Answer 2

If you're writing lexer grammar which wouldn't be used across multiple parser grammmar(s) then you can ignore this warning shown by ANTLRWorks2.

Vedetta

I'm writing a combined grammar (and I love it, much better than LEX + YACC), but I want the gramar to create an AST, which will be consumed by at least two tree grammars. Does that count? Also, why the downvote? – Renegado

I disagree with this answer because this warning can alert developers to grammar problems even if the tokens are only used in one grammar. I gave several examples in my answer. – Sergiosergipe

Answer 3

This could be a spurious error. In that case, if using IntelliJ, then the fix is to right-click the parser file and select Generate ANTLR Recognizer.

Ballesteros