assembly-merge-strategy issues using sbt-assembly

Question

I am trying to convert a scala project into a deployable fat jar using sbt-assembly. When I run my assembly task in sbt I am getting the following error:

Merging 'org/apache/commons/logging/impl/SimpleLog.class' with strategy 'deduplicate'
    :assembly: deduplicate: different file contents found in the following:
    [error] /Users/home/.ivy2/cache/commons-logging/commons-logging/jars/commons-logging-1.1.1.jar:org/apache/commons/logging/impl/SimpleLog.class
    [error] /Users/home/.ivy2/cache/org.slf4j/jcl-over-slf4j/jars/jcl-over-slf4j-1.6.4.jar:org/apache/commons/logging/impl/SimpleLog.class

Now from the sbt-assembly documentation:

If multiple files share the same relative path (e.g. a resource named application.conf in multiple dependency JARs), the default strategy is to verify that all candidates have the same contents and error out otherwise. This behavior can be configured on a per-path basis using either one of the following built-in strategies or writing a custom one:

• MergeStrategy.deduplicate is the default described above
• MergeStrategy.first picks the first of the matching files in classpath order
• MergeStrategy.last picks the last one
• MergeStrategy.singleOrError bails out with an error message on conflict
• MergeStrategy.concat simply concatenates all matching files and includes the result
• MergeStrategy.filterDistinctLines also concatenates, but leaves out duplicates along the way
• MergeStrategy.rename renames the files originating from jar files
• MergeStrategy.discard simply discards matching files

Going by this I setup my build.sbt as follows:

import sbt._
import Keys._
import sbtassembly.Plugin._
import AssemblyKeys._
name := "my-project"
version := "0.1"
scalaVersion := "2.9.2"
crossScalaVersions := Seq("2.9.1","2.9.2")

//assemblySettings
seq(assemblySettings: _*)

resolvers ++= Seq(
    "Typesafe Releases Repository" at "http://repo.typesafe.com/typesafe/releases/",
    "Typesafe Snapshots Repository" at "http://repo.typesafe.com/typesafe/snapshots/",
    "Sonatype Repository" at "http://oss.sonatype.org/content/repositories/releases/"
)

libraryDependencies ++= Seq(
    "org.scalatest" %% "scalatest" % "1.6.1" % "test",
    "org.clapper" %% "grizzled-slf4j" % "0.6.10",
    "org.scalaz" % "scalaz-core_2.9.2" % "7.0.0-M7",
    "net.databinder.dispatch" %% "dispatch-core" % "0.9.5"
)

scalacOptions += "-deprecation"
mainClass in assembly := Some("com.my.main.class")
test in assembly := {}
mergeStrategy in assembly := mergeStrategy.first

In the last line of the build.sbt, I have:

mergeStrategy in assembly := mergeStrategy.first

Now, when I run SBT, I get the following error:

error: value first is not a member of sbt.SettingKey[String => sbtassembly.Plugin.MergeStrategy]
    mergeStrategy in assembly := mergeStrategy.first

Can somebody point out what I might be doing wrong here?

Thanks

Floriaflorian

mergeStrategy is deprecated now. Try instead assemblyMergeStrategy. I'm using assembly version 0.14.10 – Bowlds

Answer 1

I think it should be MergeStrategy.first with a capital M, so mergeStrategy in assembly := MergeStrategy.first.

Prefabricate

Slapping my head. That was such an oversight. Thanks for pointing it out. – Floriaflorian

doesn't work for me, it says: build.sbt:27: error: not found: value mergeStrategy mergeStrategy in assembly := MergeStrategy.first ^ [error] Type error in expression – Dancer

@gjain no, i had to basically copy the entire code and change how the mergeStrategy resolves collisions by accepting the first version instead of failing out (aka what maven does): gist.github.com/stantonk/5303b7ec84b782a58628 I don't know why sbt doesnt do this... – Dancer

This is not correct. according to the documentation. You can't do mergeStrategy in assembly := MergeStrategy.first because mergeStrategy in assembly expects a function and you'll get a type mismatch error – Pyrolysis

Answer 2

As for the current version 0.11.2 (2014-03-25), the way to define the merge strategy is different.

This is documented here, the relevant part is:

NOTE: mergeStrategy in assembly expects a function, you can't do

mergeStrategy in assembly := MergeStrategy.first

The new way is (copied from the same source):

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
    case "application.conf" => MergeStrategy.concat
    case "unwanted.txt"     => MergeStrategy.discard
    case x => old(x)
  }
}

This is possibly applicable to earlier versions as well, I don't know exactly when it has changed.

Asia

The last example (which is official) doesn't work on scala 10, showing Type mismatch – Cuculiform

@Cuculiform I have tested the snippet in an assembly.sbt, and it works (Scala 2.10.3, sbt 0.13.2). – Asia

Answer 3

this is the proper way to merge most of the common java/scala projects. it takes care of META-INF and classes.

also the service registration in META-INF is taken care of.

assemblyMergeStrategy in assembly := {
case x if Assembly.isConfigFile(x) =>
  MergeStrategy.concat
case PathList(ps @ _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
  MergeStrategy.rename
case PathList("META-INF", xs @ _*) =>
  (xs map {_.toLowerCase}) match {
    case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
      MergeStrategy.discard
    case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
      MergeStrategy.discard
    case "plexus" :: xs =>
      MergeStrategy.discard
    case "services" :: xs =>
      MergeStrategy.filterDistinctLines
    case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
      MergeStrategy.filterDistinctLines
    case _ => MergeStrategy.first
  }
case _ => MergeStrategy.first}

Lucre

+1 for the META-INF rules. That helped me fix an issue with packaging jpmml-spark together with other libs into one jar. – Searcy

Agreed with Elmar – thank you! The accepted answer more directly addresses OP's issue, but these META-INF rules are gold; most suggestions (on this page and elsewhere) discard everything under META-INF but this loses needed config for many libraries. – Illimitable

Answer 4

I have just setup a little sbt project that needs to rewire some mergeStrategies, and found the answer a little outdated, let me add my working code for versions (as of 4-7-2015)

• sbt 0.13.8
• scala 2.11.6

• assembly 0.13.0

  mergeStrategy in assembly := {
    case x if x.startsWith("META-INF") => MergeStrategy.discard // Bumf
    case x if x.endsWith(".html") => MergeStrategy.discard // More bumf
    case x if x.contains("slf4j-api") => MergeStrategy.last
    case x if x.contains("org/cyberneko/html") => MergeStrategy.first
    case PathList("com", "esotericsoftware", xs@_ *) => MergeStrategy.last // For Log$Logger.class
    case x =>
       val oldStrategy = (mergeStrategy in assembly).value
       oldStrategy(x)
  }
  

Musky

mergeStrategy is deprecated now, use assemblyMergeStrategy instead. – Intrados

Answer 5

For the new sbt version (sbt-version :0.13.11), I was getting the error for slf4j; for the time being took the easy way out : Please also check the answer here Scala SBT Assembly cannot merge due to de-duplication error in StaticLoggerBinder.class where sbt-dependency-graph tool is mentioned which is pretty cool to do this manually

assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) {
  (old) => {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
}

Hindustan

Answer 6

Quick update: mergeStrategy is deprecated. Use assemblyMergeStrategy. Apart from that, earlier responses are still solid

Inevitable

Answer 7

Add following to build.sbt to add kafka as source or destination

 assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 //To add Kafka as source
 case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => 
 MergeStrategy.concat
 case x => MergeStrategy.first
 }

Bays