Unable to invoke avro-maven plugin

Question

My question is similar to Unable to compile and create .avro file from .avsc using Maven

I have tried all possible things, checked the maven project 100 times, still i am not able to run the avro-maven plugin to generate the code for my avsc file.

i have read the following posts and followed the same, but to no success http://grepalex.com/2013/05/24/avro-maven/ https://github.com/phunt/avro-maven-plugin

i downloaded the above maven project, and here also the result is same.

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Avro Maven Example 0.0.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ avro-maven ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ avro-maven ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory F:\01_Work\FLink\avro-maven-master\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ avro-maven ---
[INFO] No sources to compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.932 s
[INFO] Finished at: 2015-08-10T19:16:44+05:30
[INFO] Final Memory: 6M/16M
[INFO] ------------------------------------------------------------------------

But no generated code.

I strongly feel Maven is my mortal enemy and i will never be able to do any work with apache projects just because i cannot get maven to work. perhaps i should consider going back to saner world of C/C++ where it doesnt require an internet connection to compile my source.

Torero

Answer 1

I publish a simple demo which is fully tested and 100% works https://github.com/xmeng1/avro-maven-demo. There are two important things for generating code by using the Avro

1. The configuration: if we want to generate code when mvn compile or mvn package, we can put configuration under the execution. If we want to generate code when running goal mvn avro:scheme, we need put the configuration to the plugin directly. (the demo includes two type configuration)
2. The namespace of the scheme file which will decide which package the generate code will belong to. {"namespace": "com.xx.xx.demo", "name": "Foo"}, the Foo.java will be created under the package com.xx.xx.demo

Catalinacatalo

Answer 2

That eclipse error in pom file doesn't matters. Make sure that your .avsc file has namespace value, where actual file is getting generated.

{
"namespace": "com.hadoop.practice.avro",
"type": "record",
"name": "StringPair",
"doc": "A pair of strings",
"fields": [
    {"name": "left", "type": "string"},
    {"name": "right", "type": "string"}
]

}

StringPair.java get generated under this namespace defined package

Benthamism

Please add some explenation to your answer. – Affectionate

Following namespace, and naming file will do – Grammatical

Answer 3

I've struggled with it for some time yesterday. Actual reason was my mistake: default pom.xml configuration which I got by calling mvn archetype:generate had all <plugin/> tags inside <plugin-management/> section, while I had to add my <plugin/> tag just inside <plugins> root! That was my mistake, and it was really simple to fix, but very tricky to understand while you see lots of plugins defined nearby all in different way!

So finally it had to look like (pay attention to <plugin-management> with lots of pre-defined plugins!):

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.8.2</version>
        <configuration>
          <!--The Avro source directory for schema, protocol and IDL files.-->
          <sourceDirectory>${project.basedir}/src/main/resources/</sourceDirectory>
          <!--The directory where Avro writes code-generated sources. IMPORTANT!! -->
          <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
        </configuration>
        <executions>
          <execution>
            <id>avro-schemas</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>schema</goal>
              <goal>protocol</goal>
              <goal>idl-protocol</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
      <plugins>
        <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
        <plugin>
          <artifactId>maven-clean-plugin</artifactId>
          <version>3.1.0</version>
        </plugin>
        <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
        <plugin>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
        </plugin>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.22.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-install-plugin</artifactId>
          <version>2.5.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-deploy-plugin</artifactId>
          <version>2.8.2</version>
        </plugin>
        <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
        <plugin>
          <artifactId>maven-site-plugin</artifactId>
          <version>3.7.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-project-info-reports-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>

Tacklind

Account created only to thanks your answer: Thanks – Duffer

Answer 4

Below is a sample POM file that I've successfully used. My guess is that your sourceDirectory & outputDirectory tags weren't properly defined...

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>helloworld</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <!-- Keep Hadoop versions as properties to allow easy modification -->
        <hadoop.version>2.6.0-cdh5.4.0</hadoop.version>
        <avro.version>1.7.7</avro.version>
        <mrunit.version>1.1.0</mrunit.version>
        <!-- Maven properties for compilation -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-tools</artifactId>
            <version>${avro.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.mrunit</groupId>
            <artifactId>mrunit</artifactId>
            <version>${mrunit.version}</version>
            <classifier>hadoop2</classifier>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <!-- Set the Java target version to 1.7 -->
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-maven-plugin</artifactId>
                <version>${avro.version}</version>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>schema</goal>
                        </goals>
                        <configuration>
                            <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
                            <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-eclipse-plugin</artifactId>
                <version>2.9</version>
                <configuration>
                    <downloadJavadocs>true</downloadJavadocs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Kendyl

my pom is identical to the one shared by you. except that i didnt use hadoop-client, and mrunit which i think are not required. the source and output directory are configured in the same way – Torero

how do i check if the project.basedir is set properly or not? can i print the value somehow? – Torero

Ok, i was able to invoke the avro plugin by invoking simply 'mvn compile' after i made a slight change to the pom. the <plugins\> tag i had put under <pluginsManagement/> tag. otherwise the eclipse gives error "Plugin execution not covered by lifecycle configuration: org.apache.avro:avro-maven-plugin:1.7.6:schema (execution: abc, phase: generate-sources)" after removing the <pluginManagement/> tag the plugin is getting invoked from command line – Torero

why the two things are contradicting each other? if i remove the <pluginManagement/> tag, why eclipse gives error about pllugin not covered under lifecycle management? if i remove the tag, the plugin itself fails to be invoked? why/? Maven world is really strange – Torero

I'm not sure. Another StackOverflow topic addressed it: #6352708 However, I use IntelliJ instead of Eclipse, so I am not sure why Eclipse would be behave that way. I stopped using Eclipse years ago because it had too many issues. – Kendyl

Answer 5

As mentioned in comments , I put a surrounding <pluginManagement> tag over <plugins> and it resolved issue for me. I am using eclipse Mars. Example :

<pluginManagement>
            <plugins>
                <plugin>....</plugin>
            </plugins>
<pluginManagement>

Reborn