Scala SBT Assembly cannot merge due to de-duplication error in StaticLoggerBinder.class
Asked Answered
S

2

11

My problem is I can no longer use the sbt-assembly plugin because some kind of dependency merge problem creeped in, between a couple people working on this project.

The problem when I run 'sbt assembly' :

[error] 3 errors were encountered during merge java.lang.RuntimeException: deduplicate: different file contents found in the following: /Users/aris.vlasakakis/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:org/slf4j/impl/StaticLoggerBinder.class /Users/aris.vlasakakis/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar:org/slf4j/impl/StaticLoggerBinder.class deduplicate: different file contents found in the following: /Users/aris.vlasakakis/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:org/slf4j/impl/StaticMDCBinder.class /Users/aris.vlasakakis/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar:org/slf4j/impl/StaticMDCBinder.class deduplicate: different file contents found in the following: /Users/aris.vlasakakis/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:org/slf4j/impl/StaticMarkerBinder.class /Users/aris.vlasakakis/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar:org/slf4j/impl/StaticMarkerBinder.class at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140) at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25) at sbtassembly.Assembly$.x$1$1(Assembly.scala:23) at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23) at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23) at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67) at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)

... and so on

I am using SBT-assembly 0.13.0, and here is the build.sbt

name := "metamorphosis"

version := "0.10.0"

scalaVersion := "2.10.4"

lazy val common = RootProject(file("../"))

val main = Project(id = "bridge", base = file(".")).dependsOn(common)

//excludeDependencies += "org.apache.logging.log4j"

resolvers ++= Seq(
//  "Akka Snapshot Repository" at "http://repo.akka.io/snapshots/",
  "Typesafe Releases" at "http://repo.typesafe.com/typesafe/releases/",
  "Sonatype OSS Releases"  at "http://oss.sonatype.org/content/repositories/releases/"
)

libraryDependencies ++= Seq(
  "com.github.nscala-time" %% "nscala-time" % "1.8.0",
  "com.sclasen" %% "akka-kafka" % "0.0.10" % "compile",
  "com.typesafe.akka" %% "akka-actor" % "2.3.2",
  "org.codehaus.groovy" % "groovy" % "2.3.7",
  "com.typesafe.scala-logging" %% "scala-logging-slf4j" % "2.1.2",
  "ch.qos.logback" % "logback-classic" % "1.1.2",
  "ch.qos.logback.contrib" % "logback-json-classic" % "0.1.2",
  "ch.qos.logback.contrib" % "logback-jackson" % "0.1.2",
  "org.slf4j" % "slf4j-api" % "1.7.7",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.2.2",
  "org.clapper" %% "argot" % "1.0.3",
  "com.typesafe" % "config" % "1.2.1",
  "net.ceedubs" %% "ficus" % "1.0.1",
  "com.typesafe.play" %% "anorm" % "2.3.6",
  "org.json4s" %% "json4s-native" % "3.2.10",
  "org.json4s" %% "json4s-jackson" % "3.2.10",
  "com.github.tototoshi" %% "scala-csv" % "1.1.2",
  "org.scalatest" %% "scalatest" % "2.2.2",
  "junit" % "junit" % "4.11",
  "org.apache.kafka" %% "kafka" % "0.8.1.1"
    exclude("javax.jms", "jms")
    exclude("com.sun.jdmk", "jmxtools")
    exclude("com.sun.jmx", "jmxri")
    exclude("org.slf4j", "slf4j-simple")
)

mainClass in assembly := Some("com.company.kafka.agent.MetamorphosisActor")

Any clues on how I can even figure out how to fix this merge problem?

Thank you!

Shaky answered 25/3, 2015 at 0:52 Comment(0)
S
14

Previous answer gives all the required feedback, but does not actually give any solution or steps to try out.

My error:

[error] (project/*:assembly) deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar:org/slf4j/impl/StaticMDCBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMDCBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar:org/slf4j/impl/StaticMarkerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMarkerBinder.class

Recommended way

track down conflicting dependency. For example, I do not want those classes:

org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticLoggerBinder.class
org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMDCBinder.class
org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMarkerBinder.class

Call sbt (and make sure to use sbt-dependency-graph plugin):

whatDependsOn org.slf4j slf4j-log4j12 1.7.10

Returns a list:

[info] org.slf4j:slf4j-log4j12:1.7.10
[info]   +-org.apache.hadoop:hadoop-auth:2.8.0
[info]   | +-org.apache.hadoop:hadoop-common:2.8.0
[info]   |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]   |   
[info]   +-org.apache.hadoop:hadoop-common:2.8.0
[info]   | +com.github.atais:test_2.11:0.0.3 [S]
[info]   | 
[info]   +-org.apache.zookeeper:zookeeper:3.4.6
[info]     +-org.apache.curator:curator-client:2.7.1
[info]     | +-org.apache.curator:curator-framework:2.7.1
[info]     | | +-org.apache.curator:curator-recipes:2.7.1
[info]     | | | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     | | |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     | | |   
[info]     | | +-org.apache.hadoop:hadoop-auth:2.8.0
[info]     | |   +-org.apache.hadoop:hadoop-common:2.8.0
[info]     | |     +-com.github.atais:test_2.11:0.0.3 [S]
[info]     | |     
[info]     | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |   
[info]     +-org.apache.curator:curator-framework:2.7.1
[info]     | +-org.apache.curator:curator-recipes:2.7.1
[info]     | | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     | |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     | |   
[info]     | +-org.apache.hadoop:hadoop-auth:2.8.0
[info]     |   +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |     +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |     
[info]     +-org.apache.curator:curator-recipes:2.7.1
[info]     | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |   
[info]     +-org.apache.hadoop:hadoop-auth:2.8.0
[info]     | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |   
[info]     +-org.apache.hadoop:hadoop-common:2.8.0
[info]       +-com.github.atais:test_2.11:0.0.3 [S]

I need to track down hadoop-common dependency since it is the one that connects com.github.atais:test_2.11:0.0.3 to the unwanted org.slf4j:slf4j-log4j12:1.7.10

And modify it:

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.8.1" exclude("org.slf4j", "slf4j-log4j12")

Easy way

simply add the merging strategy flag for conflicting paths:

assemblyMergeStrategy in assembly := {
  ...
  case PathList("org", "slf4j", xs@_*) => MergeStrategy.first
  case x => (assemblyMergeStrategy in assembly).value(x)
}
Strife answered 19/7, 2017 at 12:56 Comment(1)
Thanks, you're the hero we don't deserveCumulative
G
8

The issue is that more than one dependency brings in slf4j transitive dependency which causes a conflict. The best way is to exclude that transitive dependency instead of using merging strategy of SBT assembly to resolve the conflict.

First you need to visualize where slf4j comes from (for all top level dependencies). You can use a tool like this sbt-dependency-graph to get a visual diagram of all dependencies. Alternatively you can explore SBT settings from SBT command prompt or use a maven site like this one or this one.

Once you locate duplicate dependencies use exclude the same way as you use in your build file. Leave only single slf4j transitive dependency. You can also exclude all slf4j transitive dependencies (see here how) and manually add that slf4j dependency at top level.

Note that if your dependencies require multiple different versions of slf4j it might be problematic to pick a single version that works for all cases. Depends on binary compatibility.

Goulet answered 25/3, 2015 at 3:32 Comment(2)
Why however, is this not an issue for the same project in sbt run, but only in the packaging of the project?Salvadorsalvadore
@matanster when SBT packages it explicitly checks for conflicts, but when it simply runs it just uses the classpath. In JVM there is no exception for such cases - first class on the path wins.Kass

© 2022 - 2024 — McMap. All rights reserved.