How to deduplicate beanutils dependencies with commons-beanutils?

Question

I have a project with several dependencies, which ultimately lead to depending on the following (I got these from sbt-dependency-graph plugin):

• commons-beanutils:commons-beanutils:1.7.0
• commons-beanutils:commons-beanutils-core:1.8.0

As a consequence, when I try to build a fat JAR using sbt-assembly, it fails with deduplication errors as:

[error] deduplicate: different file contents found in the following:
[error] /Users/someuser/.ivy2/cache/commons-beanutils/commons-beanutils/jars/someuser-beanutils-1.7.0.jar:org/apache/commons/beanutils/BasicDynaBean.class
[error] /Users/someuser/.ivy2/cache/commons-beanutils/commons-beanutils-core/jars/commons-beanutils-core-1.8.0.jar:org/apache/commons/beanutils/BasicDynaBean.class

Since I need both dependencies, I tried to shade one of them using the following rule:

ShadeRule.rename("org.apache.commons.beanutils.**" -> "shadedstuff.beanutils.@1").inLibrary("commons-beanutils" % "commons-beanutils" % "1.7.0").inAll

But then I get the following error:

[error] deduplicate: different file contents found in the following:
[error] /Users/someuser/.ivy2/cache/commons-beanutils/commons-beanutils/jars/someuser-beanutils-1.7.0.jar:shadedstuff/beanutils/BasicDynaBean.class
[error] /Users/someuser/.ivy2/cache/commons-beanutils/commons-beanutils-core/jars/commons-beanutils-core-1.8.0.jar:shadedstuff/beanutils/BasicDynaBean.class

As if the shading process is applied to both artifacts. How to shade a specific artifact?

Valdavaldas

For the shade plugin see #14403245 – Dubonnet

Answer 1

Since I need both dependencies, I tried to shade one of them using the following rule:

ShadeRule.rename("org.apache.commons.beanutils.**" -> "shadedstuff.beanutils.@1").inLibrary("commons-beanutils" % "commons-beanutils" % "1.7.0").inAll

Shading is a contributed feature on sbt-assembly, which basically integrates with Jar Jar Links, so I'm not expert on this.

Having said that, I think .inAll defeats the purpose of your .inLibrary(...) call. Maybe the way I described it on the README wasn't clear enough.

I think what you want to do is to put inLibrary(...) for commons-beanutils 1.7.0 and all of its callers who is referring to commons-beanutils 1.7.0 code as org.apache.commons.beanutils.** (in this case Hadoop?)

Misbeliever

Indeed. Removing .inAll fixes the issue. I misunderstood the documentation, but now everything seems pretty clear. Thanks a lot for the support. – Valdavaldas

@ale64bit, Can you please a snippet of the working build.sbt file. I have a similar issue while depending org.apache.hadoop:hadoop-aws:2.7.3. – Piggish