sbt assembly error:deduplicate: different file contents found in io.netty.versions.properties

Question

I have added the following jar to build.sbt file as follows:

"com.amazonaws" % "aws-java-sdk" % "1.11.492"

Post this ,during merge , I am getting the following error :

 [error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/jenkins-slave/.ivy2/cache/io.netty/netty-codec-http/jars/netty-codec-http-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-resolver/jars/netty-resolver-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.1.17.Final.jar:META-INF/io.netty.versions.properties
    at sbtassembly.Assembly$.applyStrategies(Assembly.scala:143)
    at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
    at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)...........

I have tried many workarounds provided for this like:

1) added this line in assemblyMergeStrategy in build.sbt:case PathList("io", "netty", xs @ _*) => MergeStrategy.discard(tried with .last and .first)

2)added this line in assemblyMergeStrategy in build.sbt:case "META-INF\\io.netty.versions.properties" =>MergeStrategy.first(tried with .last and .discard)

3)added SBT exclusion rules for the errored out netty jars in excludedDependencies like below:

    excludeDependencies ++= Seq(
      SbtExclusionRule("io.netty", "netty-codec-http"),
      SbtExclusionRule("io.netty", "netty-codec"),
      SbtExclusionRule("io.netty", "netty-handler"),
      SbtExclusionRule("io.netty", "netty-transport"),
      SbtExclusionRule("io.netty", "netty-resolver"),
      SbtExclusionRule("io.netty", "netty-buffer"),
      SbtExclusionRule("io.netty", "netty-common")
    )

and many such variations of the above.None of these solutions are working.

plugins.sbt looks like below:

addSbtPlugin("io.spray" % "sbt-revolver" % "0.7.2")

addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.5.0")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.0.1")

addSbtPlugin("com.scalapenos" % "sbt-prompt" % "0.2.1")

addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.1.10")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.0")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

Kindly advise.

Flyman

Have you tried "com.amazonaws" % "aws-java-sdk" % "1.11.492" excludeAll ExclusionRule(organization = "io.netty")? In our build it is case "META-INF/io.netty.versions.properties" => MergeStrategy.first btw – Ursulaursulette

Thank you @ariskk, I tried this and it worked!!! – Flyman

Answer 1

You could try below:

assemblyMergeStrategy in assembly := {
      case x if x.contains("io.netty.versions.properties") => MergeStrategy.discard
      case x =>
        val oldStrategy = (assemblyMergeStrategy in assembly).value
        oldStrategy(x)
}

or in worst case

case x if x.contains("versions.properties") => MergeStrategy.discard

Empurple

Answer 2

If you check the contents of the file you can see that every jar has different values in it, so to be correct in the assembly you have concat these files to preserve all the properties in them. Use case "META-INF/io.netty.versions.properties" => MergeStrategy.concat it the strategy.

Lithology

Answer 3

I've run into the same issue on windows, and found the solution should be:

case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.first

or putting it in the assemblyMergeStrategy settings, it will be:

assembly / assemblyMergeStrategy := {
  case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.first
  case x =>
    val oldStrategy = (assembly / assemblyMergeStrategy).value
    oldStrategy(x)
}

---

I think the reasons why the workarounds mentioned in the question doesn't work is because:

1. the io.netty.versions.properties is a file under META-INF, instead of a file named versions.properties under io/netty. So this one doesn't matched the intended file:

case PathList("io", "netty", xs @ _*) => MergeStrategy.discard

2. I think the reason why the second one doesn't work is because we need to use PathList to match against a path under some folders, instead of using slashes like it shows in the question(with a backslash):

case "META-INF\\io.netty.versions.properties" =>MergeStrategy.first

When I first ran into this issue, I was using the forward slash, which doesn't work either:

case "META-INF/io.netty.versions.properties" =>MergeStrategy.first

3. I think the netty dependencies are required for the aws-java-sdk to work. So excluding them from the dependencies will get us ClassNotFoundException. Which will not solve the problem.

Mcbroom

Answer 4

After searching for a solution to the problem I found this issue on the sbt-assembly repository.

It contains the following workaround that solved the problem for me:

It works when the external dependency is replaced with an earlier version of the aws library, "com.amazonaws" % "aws-java-sdk" % "1.11.80", because that one does not contain any duplicates of META-INF/io.netty.versions.properties.

I can imagine this not working for the ones who need the very last version, but this is at least something.

Sachsse

Answer 5

I think you are getting this error because of multiple versions availability. To get away from this you can try the below snippet in your build.sbt and try.

assemblyShadeRules in assembly := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)

Tribal