Maven build [WARNING] we have a duplicate class

Question

Anybody has any idea what happened to my maven build? I am getting a lot of duplicate warnings.

[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/NoOpLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/Jdk14Logger.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar

I've looked into my local m2 repo, I have two classes there in commons-logging-api jar, LogFactoryImpl.class and LogFactoryImpl$1.class. Same as all the classes mentioned in the warnings.

One thing to mention is that I am using shade plugin in my pom.xml.

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.4</version>
            <configuration>
                <createDependencyReducedPom>true</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.~~~~black out my own main class here~~~~~</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>

I noticed that the dependency tree looks like as below

[INFO] +- org.apache.cxf:cxf-bundle-jaxrs:jar:2.5.1:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] \- org.apache.hadoop.hive:hive-jdbc:jar:0.7.1-cdh3u3:compile
[INFO]    \- org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile
[INFO]       \- commons-logging:commons-logging-api:jar:1.0.4:compile

and commons-logging.jar and commons-logging-api.jar both have org/apache/commons/logging/LogFactory.class.

somehow Shad plugin is trying to squeeze them in to a big fat jar at the end. then the warning is showing up. It's been said this is ignorable warning. But I am a bit worried, How does the application know what is the exact class should be used if there are two duplicated class with the same name?

Statuesque

LogFactoryImpl.class and LogFactoryImpl$1.class the class with $1 in the name is local class inside LogFactoryImpl. – Vermiculite

Answer 1

Take a look at the "Dependency Exclusions" section in the Maven doc.

In your provided example, I'll exclude the commons-logging:commons-logging-api:jar:1.0.4:compile dependency from org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile. In your pom.xml :

    <dependency>
        <groupId>org.apache.hadoop.hive</groupId>
        <artifactId>hive-common:jar</artifactId>
        <version>0.7.1-cdh3u3</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Applesauce

Thanks a lot, that make sense. so what's the best way to manage the dependencies like this. let's say, you have projectA.jar, projectB.jar, they have the same dependencies in different versions. eg. dependency.1.0, dependency.2.0. It has risks to just use one of them. Is there a way to let ProjectA keeps using depdency.1.0 while ProjectB still uses dependency.2.0? – Statuesque

Nope, there is no way (maybe with OSGI but I'm not an expert in this area). In this case, it is the class loader which selects the class to use (depending on the OS, JVM etc..., it is often the first jar referenced in the classpath that is used). Your best bet is to take the latest version (exclude other versions), hoping that it is backward compatible... But it is better than letting the classloader choosing for you. – Applesauce

Answer 2

You might also have run into a limitation of maven-shader-plugin. It replaces the default jar artifact (created by maven-jar-plugin). This works fine on a clean build, but on a rebuild where the jar is not regenerated, the shader runs again on the jar it created last time, which already contains copies of all the class dependencies. That produces a lot of warnings about duplicates.

This issue is still unaddressed as of maven-shader-plugin 2.0: https://issues.apache.org/jira/browse/MSHADE-126

One workaround is to add the maven-jar-plugin explicitly to your pom.xml and add the configuration setting <forceCreation>true</forceCreation>.

Cohere

This is what caused it for me. You can solve it by following the solution here: #8880861 – Scantling

Answer 3

In my case, my parent pom was including commons-beanutils and my child module (which is the only thing I wanted to compile) was including commons-io.

The shade plug in complained about duplicates since commons-io and commons-beansutil shared some common classes. Note that beansutiul was being included even though it was not needed, and was not used.

I solve this by minimizing the jar by adding this to the configuration:

<minimizeJar>true</minimizeJar>

Now the shade plugin did not add unused resources.

Warning went away.

Bindery

Answer 4

All above (about reviewing dependencies tree and excluding) is correct in the most of cases, but in my case (i didn't have overlapping in my dependencies) preliminary clean helped (don't know why though):

mvn clean package

Lionize

Answer 5

You can exclude the jar you don't want (the ones that are giving the duplicate warnings using the following tags under the shade plugin -

<configuration>
    <artifactSet>
        <excludes>
            <exclude>commons-logging:commons-logging</exclude>
        </excludes>
    </artifactSet>
    <minimizeJar>true</minimizeJar>
</configuration>

More details can be found at http://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html

Laurellaurella

Answer 6

I saw this happen in eclipse when I updated my parent project's dependencies.

I deleted all the files in my target directory and it fixed the issues.

Blueness

Answer 7

In my case I was relying on a package that also creates a shaded jar.

Shaded jars are meant for deployment, not installing as a dependency.

Creating a reduced dependency POM during the build process of the dependency, instructs maven on which dependencies can be left out.

In the maven-shade-plugin configuration:

<configuration>
  <createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>

For more details see this post:

What is the maven-shade-plugin used for, and why would you want to relocate java packages?

The error I was getting from maven:

WARNING: x.jar, y.jar contain overlapping classes

Schoening

Answer 8

You have dependencies in your pom which contain duplicate classes but without the appropriate pom i couldn't say a word about it.

Vermiculite

added more info, can you have a look at it again? Thanks – Statuesque

Have you checked which of the dependencies contains the duplicate classes? Based on your output the commons-logging-api-1.0.4.jar which seemed to be coming from using a different api version (duplicated) or you are using an other logging provider (slf4j ?). – Vermiculite