Generating ANTLR4 grammar files with package declaration in gradle

Question

I try to use the gradle antlr plugin, but run into several problems.

I have a grammar file called wls.g4:

grammar WlsScript;

@header {
   package hu.pmi.wls.antlr.ws;
} 

program
  : 'statementList'? EOF
  ;

// Several more grammar and lexer rules

(Note: I made the statementList to a keyword only to make a correct grammer without including the whole grammar. ;-))

This file is located in /src/main/antlr (as the default source folder of the antlr production grammar files).

Here is the snippet from build.gradle:

project('common') {

    apply plugin: 'antlr'

    dependencies {
       // Some dependencies

       antlr "org.antlr:antlr4:4.5"
    }
} 

When I use the generateGrammarSource gradle task (comming from antlr plugin) to generate the source files it generates the files in build/generated-src/antlr/main folder which is the default.

What goes wrong, that it doesn't create the folders of the java package (hu/pmi/wls/antlr/ws in our case) so the source will be incorrectly located in Eclipse.

My primary question is how could I force the task to generate source files in a package-structured way? In other words, how can I configure the gradle task to use the package declaration from grammar?

Thanks!

Quietism

I am having the same issue with ANTLR 4.5. I am only using the org.antlr.v4.Tool to process the .g4 file and couldn't get it to generate the Java source in their corresponding package name folder structure. It could generate files according to the directory specified by the "-o" option. So the only way is to specify explicitly the package folder structure in the "-o" option. I am not familiar with Gradle but lookout if there's a way for Gradle to pass in options while executing the ANTLR tool. – Eglanteen

Answer 1

I had the exact same question and I was able to change the outputDirectory used by the AntlrTask to get the package name in there.

generateGrammarSource {
    outputDirectory = new File("${project.buildDir}/generated-src/antlr/main/net/example".toString())
}

I found no easy way to extract the grammar package from the files so had to hardcode it in the task.

Denouement

How do you reconcile this in an IDE? I could write something like import whatever.gen.MyParser and MyParser x = new MyParser(...), but the ide screams away about how it cant find source for that --because the source is 'just in time', for lack of a better phrase. Maybe its better to have a gen-src folder thats acknowledged by the IDE but ignored under version control? is that antithetical to gradle conventions? – Theocritus

You can add the generated-src directory as a source directory in IDE (I'm using IntelliJ) – Denouement

Answer 2

Add this to your sub-project build.gradle

generateGrammarSource {
    outputDirectory = file("src/main/java/com/example/parser")
}

add this to your grammar after your "grammar ";

@header {
    package com.example.parser;
}

Tested and working with Java8 grammar from antlr example grammars

Additional Link(s):

Here is a short guide of the Antlr plugin from docs.gradle.org

I haven't tested this while putting it in parent-project like it looks like you did, but should be easy enough to extend to do that. (I try to keep parent project empty other than repositories to use.)

Reinaldo

This answer suggests that setting the outputDirectory into your source code like this is dangerous because a gradle 'build clean' can then wipe out all source files in that directory. The answer suggests moving the generated files via gradle's 'copy' command to their future locations AFTER they are generated into generated-src. – Purlieu

Answer 3

Transfer a generated ANTLR files to the project folder and ZZZ subfolder.

antlr.g4

....

@header {
package XXX.YYY.ZZZ;
}

.....

build.gradle

group 'XXX'
version '0.0.1'

apply plugin: 'java'
apply plugin: 'application'
apply plugin: 'antlr'
apply plugin: 'eclipse'

mainClassName = 'XXX.YYY.Test'
final antlr4Version = '4.7'
final antlrGen = 'gen'
final baseDir = 'src/main/java'

repositories {
    mavenCentral()
}

dependencies {
    antlr "org.antlr:antlr4:"+antlr4Version
}

generateGrammarSource {
    outputDirectory = file(antlrGen)
    maxHeapSize = '64m'
    arguments += ['-visitor', '-long-messages']
}

generateGrammarSource.doLast {
    moveAntlrGeneratedFilesToTheirPackages(source, antlrGen, baseDir)
}

def moveAntlrGeneratedFilesToTheirPackages(FileTree grammarFiles, generatedFolder, destFolder) {
    grammarFiles.each {File file ->
        final grammarName = file.name.lastIndexOf('.') >= 0 ? file.name[0 .. file.name.lastIndexOf('.') - 1] : file.name
        final grammarPackage = extractPackageNameFromGrammerFile(file)
        copy {
            from generatedFolder
            include "${grammarName}*.*"
            into destFolder + "/" + grammarPackage.replaceAll("\\.", "/")
        }
    }
    project.delete fileTree(generatedFolder).include('*.*')
}

def extractPackageNameFromGrammerFile(File grammarFile) {
    def grammarPackage = "unknown.package"
    def packageRegex = ~/[ ]*package[ ]*([a-zA-Z]+[a-zA-Z0-9.-_]*)[ ]*;/
    grammarFile.eachLine { line ->
        def matcher = packageRegex.matcher(line)
        if (matcher.find()){
            grammarPackage = matcher.group(1)
        }
    }
    return grammarPackage
}

jar {
    baseName = 'XXX.YYY'
    version =  '1.0.0'
    manifest.attributes('Main-Class': mainClassName);
}

task wrapper(type: Wrapper) {
}

Test.java

package XXX.YYY;

import java.lang.Exception;
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.tree.ParseTree;
import XXX.YYY.ZZZ.*;

public static void main(String[] args) throws Exception {

...

Coursing

Answer 4

I have similar issue using antlr3. You can fix the positions appending a post processing to the generateGrammarSource task:

/* Antlr generated files have wrong position so fix it with a post generation processing */
import groovy.io.FileType
generateGrammarSource << {
    logger.info("Antlr doesn't produce files to rigth place. Fix using package declaration")
    def move = [/*File wrong position*/:/*File rigth position*/]
    generateGrammarSource.outputDirectory.eachFileMatch( FileType.FILES, ~/.*\.java/ ) { f ->
        def pkg = f.readLines().find { it.trim() =~ /^package/ }
        if (pkg) {
            pkg = pkg.split( ' ' )[1].replace( ';','' )
            pkg = pkg.replace( '.','/' )
            move.put( f,new File("${generateGrammarSource.outputDirectory}/$pkg/${f.name}") )
        }
    }
    move.each { s,t ->
    logger.info( "Moving ${s.name} to right location .." )
    t.parentFile.mkdirs()
    s.renameTo( t.absolutePath )
}

Domingo

Answer 5

Put your grammar file in hu/pmi/wls/antlr/ws directory.

Peroneus

As per the plugin docs : " If the ANTLR grammar is organized in packages, the structure in the antlr folder should reflect the package structure. This ensures that the generated sources end up in the correct target subfolder." – Periderm