ANTLR - allow any char between to keywords

Question

I'd like to define a grammar for a simple language.

The language allows some sort of assignments.

Example

keyworda: this is the 1 keyword-A
keywordb: this is the second keywordb
...

The thing is, that after the keyword and the ':' any char should be possible (the keyword, too)

I've tried many thing but I think I'm still not that into the lexer and parser thinking...

My last idea failed:

rule 
  :  'keyworda' ':' anychar* 'keywordb' ':' anychar* EOF 
  ;

anychar
  :  .
  ;

NEWLINE
  :  ('\r'? '\n') {$channel=HIDDEN;}
  ;

EDIT

First of all: thanks for your answer!

I read through the manual and looked the tutorials by scott stanchfield.

The problem is, that I don't get the "anychar" thing!

You are right, the grammar I postet above was wrong because I was in hurry.

A better try is this ahead. Problem is still, that the Tokenizer recognizes e.g. keyworda in a definition ala

keyworda : this is keyworda.
keywordb : this is another key!
...

The grammar:

rule
    :   KEYA ':' STRING_LITERAL* NEWLINE
        keybdefinition*
         EOF
    ;

keybdefinition
    :   KEYB ':' STRING_LITERAL* NEWLINE
    ;


KEYA: 'keyworda';
KEYB:'keywordb';
STRING_LITERAL: 'a'..'z' | 'A'..'Z' | '0'..'9' | ':' | '.' | '&' | '/' | '\\' | ';';

NEWLINE: '\r'? | '\n'; 
SPACE:  (' ' | '\t') {$channel=HIDDEN;};

EDIT II

Oh my god, it's quiet obvious to do it the way you explained it. Don't know why I didn't get it myself! Great thanks Tim for your explanation!

I've got just one more question left: If I define my tokens for the lexer and my grammar for the parser. Is it the common way to check the semantic in the tree parser or in the parser itself?

For example, let's assume I have the same grammar defined as you posted.

keyworda : ab
keywordb : xy
keyworda : ab1
keywordb : xy1
...

Now I want to check if after every keyworda definition a keywordb is defined. Later I do want to check if the meaning if the value is proper. Lets assume we do have a keyword extends : 'keyword value' and I need to check if 'keyword value' is already defined.

I could do this in 2 ways: First, change your grammar rule for the parser and add java code for the checkings right there. Second the grammar stays as it is and I define a tree parser grammer to check those conditions.

I do not really know which way is the better one and what the advantages or disadvantages are...

Thanks a lot for your help

Stinkstone

Alexander, note that I've edited my answer. – Knacker

Note that Tim didn't answer your question... :). To answer your second question (EDIT II): I'd keep the parser grammar clear from embedded (Java) code as much as I can and create a tree walker just for checking "double defined" keys. – Knacker

Answer 1

The . has a different meaning inside lexer- and parser-rules. Inside lexer rules, it matches any character from the range \u000...\uFFFF. And inside parser rules, . matches any token.

Note that lexer rules start with a capital, and parser rules start with a lower case letter. You can also create tokens (lexer rules) as literals inside parser rules. That means your grammar will only create 4 diferent tokens (really 3, since NEWLINE is "hidden"):

• 'keyworda'
• ':'
• 'keywordb'
• NEWLINE (which is removed from the default token-stream)

(EOF is a built-in token)

So, that makes your anychar rule match either 'keyworda', ':' or 'keywordb', and not any character as you might expect.

Also, it seems you're separating your key ':' value-entries by line-breaks, yet you're removing line breaks during the lexing-phase. By removing them, how are you supposed to know what the end of a value is and what the start of a key is? Your token-stream would be one continues stream of keywords, any chars and colons, so there's no way to tell when a keyword really is a keyword, or part of the value (right of the ':'). You need a line-break token for that.

It looks like you've started using ANTLR without really knowing what you're doing: IMO, this is not the way to learn this specific tool. I recommend getting hold of a copy of The Definitive ANTLR Reference or reading/looking at some ANTLR tutorials before continuing.

Best of luck!

EDIT

Here's a quick demo of how to let a keyword also be a part of your "value":

file: T.g

grammar T;

parse
  :  line+ EOF
  ;

line
  :  key COLON value eol 
     {System.out.printf("key='\%s', value='\%s'\n", $key.text, $value.text);}
  ;

value
  :  any_except_newline*
  ;

key
  :  KEYA
  |  KEYB
  ;

any_except_newline
  :  COLON
  |  KEYA
  |  KEYB
  |  WORD
  |  ANYCHAR
  ;

eol
  :  NEWLINE
  |  EOF
  ;

COLON   : ':';
KEYA    : 'keyworda';
KEYB    : 'keywordb';
WORD    : ('a'..'z' | 'A'..'Z')+;
NEWLINE : '\r'? '\n' | '\r';
SPACE   : (' ' | '\t') {$channel=HIDDEN;};
ANYCHAR : .; 

file: Main.java

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    String source = 
        "keyworda : this is keyworda.\n" + 
        "keywordb : this is another key!";
    TLexer lexer = new TLexer(new ANTLRStringStream(source));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.parse();
  }
}

If you now run the demo by doing:

java -cp antlr-3.3.jar org.antlr.Tool T.g
javac -cp antlr-3.3.jar *.java
java -cp .:antlr-3.3.jar Main

you'd see to following being printed to the console:

key='keyworda', value='this is keyworda.'
key='keywordb', value='this is another key!'

Knacker

Given that ANTLR's CharStream emits ints and not chars, I would guess that . actually matches a single code point, not a single char. If it really does match only a single char, then that is very scary indeed for anyone wishing to treat Unicode text correctly. – Longs

I'm not sure when this changed, but somewhere down the line of v3, I believe this changed from char to int. Could very well be that at the time of this answer, it was char (it's already more than 10 years ago!). Anyway, v4 is (of course) as you say int. – Knacker