How do I get an Antlr Parser rule to read from both default AND hidden channel

Question

I use the normal whitespace separation into the hidden channel but I have one rule where I would like to include any whitespace for later processing but any example I have found requires some very strange manual coding.

Is there no easy option to read from multiple channels like the option to put the whitespace there from the beginning.

Ex. this is the WhiteSpace lexer rule

WS  :   ( ' '
        | '\t'
        | '\r'
        | '\n'
        ) {$channel=HIDDEN;}
    ;

And this is my rule where I would like to include whitespace

raw :   '{'? (~('{'))*;

Basically it's a catch all rule to capture any content that does not match other rules to be processed by another pattern and therefore I need the original stream.

I was hoping for a {$channel==DEFAULT || $channel==HIDDEN} syntax example but cannot find any.

My target will be C# but I can rewrite Java examples if required.

Satanic

Answer 1

AFAIK, that is not possible. However, you could extend the UnbufferedTokenStream to change the channel during parsing. You can't use the CommonTokenStream since it buffers a variable amount of tokens (and there can be tokens in the buffer that are on the wrong channel!). Note that you need at least ANTLR 3.3: in previous versions the UnbufferedTokenStream wasn't included yet.

Let's say you want to parse (and display) either lower- or upper case letters. Upper case letters are put on the HIDDEN channel, so by deafult, only lower case letters will be parsed. However, when the parser stumbles upon a lower case "q", we want to change to the HIDDEN channel. Once parsing on the HIDDEN channel, we want the "Q" to bring us back to the DEFAULT_CHANNEL again.

So when parsing the source "aAbBcqCdDQeE", first "a", "b" and "c" are printed, then the channel is changed, then "C" and "D" get printed, then the channel is changed again, and finally "e" is printed to the console.

Here's an ANTLR grammar that does this:

ChannelDemo.g

grammar ChannelDemo;

@parser::members {
  private void handle(String letter) {
    if("Q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(Token.DEFAULT_CHANNEL);
    }
    else if("q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(HIDDEN);
    }
    else {
      System.out.println(letter);
    }
  }
}

parse
  :  any* EOF
  ;

any
  :  letter=(LOWER | UPPER) {handle($letter.getText());}
  ;

LOWER
  :  'a'..'z'
  ;

UPPER
  :  'A'..'Z' {$channel=HIDDEN;}
  ;

And here's the custom token stream class:

ChangeableChannelTokenStream.java

import org.antlr.runtime.*;

public class ChangeableChannelTokenStream extends UnbufferedTokenStream {

    public ChangeableChannelTokenStream(TokenSource source) {
        super(source);
    }

    public Token nextElement() {
        Token t = null;
        while(true) {
            t = super.tokenSource.nextToken();
            t.setTokenIndex(tokenIndex++);
            if(t.getChannel() == super.channel) break;
        }
        return t;
    }

    public void setChannel(int ch) {
        super.channel = ch;
    }
}

And a small Main class to test it all:

Main.java

import org.antlr.runtime.*;

public class Main {
    public static void main(String[] args) throws Exception {
        ANTLRStringStream in = new ANTLRStringStream("aAbBcqCdDQeE");
        ChannelDemoLexer lexer = new ChannelDemoLexer(in);
        ChangeableChannelTokenStream tokens = new ChangeableChannelTokenStream(lexer);
        ChannelDemoParser parser = new ChannelDemoParser(tokens);
        parser.parse();
    }
}

Finally, generate a lexer/parser (1), compile all source files (2) and run the Main class (3):

1

java -cp antlr-3.3.jar org.antlr.Tool ChannelDemo.g

2

javac -cp antlr-3.3.jar *.java

3 (*nix)

java -cp .:antlr-3.3.jar Main

3 (Windows)

java -cp .;antlr-3.3.jar Main

which will cause the following to be printed to the console:

a
b
c
C
D
e

EDIT

You can include the class in your grammar file like this:

grammar ChannelDemo;

@parser::members {
  private void handle(String letter) {
    if("Q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(Token.DEFAULT_CHANNEL);
    }
    else if("q".equals(letter)) {
      ((ChangeableChannelTokenStream)input).setChannel(HIDDEN);
    }
    else {
      System.out.println(letter);
    }
  }

  public static class ChangeableChannelTokenStream extends UnbufferedTokenStream {

    private boolean anyChannel;

    public ChangeableChannelTokenStream(TokenSource source) {
      super(source);
      anyChannel = false;
    }

    @Override
    public Token nextElement() {
      Token t = null;
      while(true) {
        t = super.tokenSource.nextToken();
        t.setTokenIndex(tokenIndex++);
        if(t.getChannel() == super.channel || anyChannel) break;
      }
      return t;
    }

    public void setAnyChannel(boolean enable) {
      anyChannel = enable;
    }

    public void setChannel(int ch) {
      super.channel = ch;
    }
  }
}

parse
  :  any* EOF
  ;

any
  :  letter=(LOWER | UPPER) {handle($letter.getText());}
  |  STAR                   {((ChangeableChannelTokenStream)input).setAnyChannel(true);}
  ;

STAR
  :  '*'
  ;

LOWER
  :  'a'..'z'
  ;

UPPER
  :  'A'..'Z' {$channel=HIDDEN;}
  ;

The parser that gets generated from the grammar above will enable reading from all channels when it encounters a "*". So when parsing "aAbB*cCdDeE":

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    ANTLRStringStream in = new ANTLRStringStream("aAbB*cCdDeE");
    ChannelDemoLexer lexer = new ChannelDemoLexer(in);
    ChannelDemoParser.ChangeableChannelTokenStream tokens =
        new ChannelDemoParser.ChangeableChannelTokenStream(lexer);
    ChannelDemoParser parser = new ChannelDemoParser(tokens);
    parser.parse();
  }
}

the following gets printed:

a
b
c
C
d
D
e
E

Radix

I will try this with the read from any channel solution as that is exactly what I was looking for :) Would it be possible to include the class within the .g file as I use AntlrWorks for development and would rather avoid having to compile and run externally? – Intercalate

How can this be done in C#? There is no 'UnbufferedTokenStream' in Antlr.Runtime or Antlr.Runtime.Misc – Shantel

@Tizz, I don't know, C# isn't my forte unfortunately. – Radix

@BartKiers And Java isn't mine. Well, theres some good points here, I hope I can translate, but it isnt going well. – Shantel

Answer 2

Maybe you should consider making whitespace a part of your gramar instead. But why clutter up your gramar with such unimportant information? Well, because it is NOT unimportant. The newline has meaining in certain contexts. When you want IDE support, e.g. from visual studio language server, you need to specify a language grammer without all the bells and whistles of low-level ANTLR customization.

Venturous

Answer 3

In Antler 4 I'm using a simple solution. I didn't test it in Antlr 3. It's C# but you can translate it to Java easily.

1. Change parser1.g4 as next:

   parser grammar Parser1;
   
   options { tokenVocab=Lexer1; }
   
   startRule
   @init { SetWhiteSpacesAcceptence(false); } 
       : (componentWithWhiteSpaces | componentWithoutWhiteSpaces)* EOF
   ;
   
   componentWithWhiteSpaces : { SetWhiteSpacesAcceptence(true); } 
                               component1 component2 component3 
                               { SetWhiteSpacesAcceptence(false); } 
   ;
   
   componentWithoutWhiteSpaces : component4 component5 component6 
   

2. Change lexer1.g4 as next:

   lexer grammar Lexer1;
   WS : [ \t\r\n] { if( this.IsWhiteSpacesAccepted() ) Skip(); };
   

3. Extend Parser1 class as next:

   class MyParser : Parser1
   {
       public void SetWhiteSpacesAcceptence(bool isAccept)
       {
           if (_input != null && _input.TokenSource != null)
           {
               if (_input.TokenSource is MyLexer)
               {
                   MyLexer lexer = _input.TokenSource as MyLexer;
                   if (lexer != null)
                       lexer.SetWhiteSpacesAcceptence(isAccept);
               }
           }
       }
   
       public bool IsWhiteSpacesAccepted()
       {
           if (_input != null && _input.TokenSource != null)
           {
               if (_input.TokenSource is MyLexer)
               {
                   MyLexer lexer = _input.TokenSource as MyLexer;
                   if (lexer != null)
                       return lexer.IsWhiteSpacesAccepted();
               }
           }
   
           return false;
       }
   }
   

4. Extend Lexer1 class as next:

   class MyLexer : Lexer1
   {
       private bool isWhiteSpacesAccepted;
   
       public void SetWhiteSpacesAcceptence(bool isAccept) { isWhiteSpacesAccepted = isAccept }
   
       public bool IsWhiteSpacesAccepted() { return isWhiteSpacesAccepted; }
   }
   

5. Now the Main function as next:

   static void Main()
   {
       AntlrFileStream input = new AntlrFileStream("pathToInputFile");
       MyLexer lexer = new MyLexer(input);
       UnbufferedTokenStream tokens = new UnbufferedTokenStream(lexer);
       MyParser parser = new MyParser(tokens);
   
       parser.startRule();
   }
   

Cookie

Thank you for the answer, I use C# my self but I have abandoned Antler for a hand coded lexel/parser, partly because antler had so many problems with the C# target for a while but also because antler has problems when our "language" is embedded in raw text like html or javascript. Antler (or at least as far as I could tell) is best suited for a more strict syntax. – Intercalate