How can a duplicate class be excluded from sbt assembly?

Question

We have a situation in which two dependencies have exactly the same class (because one of the dependencies copied it and included in their own source).

This is causing sbt assembly to fail its deduplication checks.

How can I exclude a class from a particular jar?

Lip

Answer 1

You need a mergeStrategy, which will take one of the files.

mergeStrategy in assembly := {
    case PathList("path", "to", "your", "DuplicatedClass.class") => MergeStrategy.first
    case x => (mergeStrategy in assembly).value(x)
}

Update

If you want to handle the file depending on the JAR which it came from, I don't think you can with the merge strategies that assembly plugin defines. What you could do you could define your own strategy.

I would invert your condition though. I think the question should be "How can I include a class from a particular JAR?". The reason is that there can be more than two JARs having the same class, and you can only include one in the end.

You can tell from where the file comes by using AssemblyUtils.sourceOfFileForMerge.

project/IncludeFromJar.scala

import sbtassembly._
import java.io.File
import sbtassembly.Plugin.MergeStrategy

class IncludeFromJar(val jarName: String) extends MergeStrategy {

  val name = "includeFromJar"

  def apply(args: (File, String, Seq[File])): Either[String, Seq[(File, String)]] = {
    val (tmp, path, files) = args
    val includedFiles = files.flatMap { f =>
      val (source, _, _, isFromJar) = sbtassembly.AssemblyUtils.sourceOfFileForMerge(tmp, f)
      if(isFromJar && source.getName == jarName) Some(f -> path) else None
    }
    Right(includedFiles)
  }

}

build.sbt

mergeStrategy in assembly := {
    case PathList("path", "to", "your", "DuplicatedClass.class") => new IncludeFromJar("jarname.jar")
    case x => (mergeStrategy in assembly).value(x)
}

Harold

Doh! Would you mind fixing <<= operator? In this case too much duplication (and scares our future users). Please... – Armalda

@JacekLaskowski I've changed it not to use <<= operator, although I don't think it's an issue. This is how it was documented. – Harold

It was indeed, but it's better so much now. Thanks for the change! People get used to the new operators sooner. – Armalda

doesn't this solution take the first class it comes across ignoring where it came from? I want to exclude a class that should not be in a jar I depend on so I need to specify that class in that jar. – Lip

@ChanningWalton I think you have to write your custom MergeStrategy to do that. Please check my updated answer. – Harold

Thanks @lpiepiora, and good point about inverting the condition. – Lip

I am getting "project\IncludeFromJar.scala:1: not found: object sbtassembly" – Marva

Have you added assembly plugin? Do you have required imports? – Harold

in project/plugins.sbt addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2"). AFA imports: I copied your code verbatim into project/IncludeFromJar.scala and intellij shows no errors but it fails from sbt command line. – Marva

Can you post a gist or similar showing full mini project usage? – Marva

Oh I had put under project/project/ instead of project. It compiles now – Marva

@javadba sure, check out on my github account – Harold

I see hundreds of "[warn] Strategy 'includeFromJar' was applied to a file" – Marva

@javadba you can adjust this by setting notifyThreshold to something bigger. This is the numbers of conflicting files that must exist for the message to be printed. For example override def notifyThreshold = 10, or whatever number works for you. – Harold

Answer 2

What version of sbtassembly are yall using? I believe Im using using a different version (0.14.2) because Im getting an error using use project/IncludeFromJar.scala.

When compiling I get the following error:

method apply cannot override final member
def apply(args: (File, String, Seq[File])): Either[String, Seq[(File, String)]] = {
^  

Upon further investigation I found out that sbtassembly.MergeStrategy's apply method is final. Therefore, IncludeFromJar's apply method cannot override sbtassembly.MergeStrategy's even with specifying override def apply in IncludeFromJar

Thanks :)

Sima

Answer 3

Probably a bit late to the discussion but to help others that find this:

when you extend MergeStrategy you want to override the method apply with the signature:

 def apply(tempDir: File, path: String, files: Seq[File]): Either[String, Seq[(File, String)]]

The apply method that is final with the tuple argument, calls the apply with the parameters split out as above

https://github.com/sbt/sbt-assembly/blob/edd35cfbaf05c3465371b63d38fda8ac579d766c/src/main/scala/sbtassembly/MergeStrategy.scala#L19

So the example becomes:

def includeFromJar(val jarName: String): sbtassembly.MergeStrategy = new sbtassembly.MergeStrategy {

  val name = "includeFromJar"

  def apply(tmp: File, path: String, files: Seq[File]): Either[String, Seq[(File, String)]] = {
    val includedFiles = files.flatMap { f =>
      val (source, _, _, isFromJar) = sbtassembly.AssemblyUtils.sourceOfFileForMerge(tmp, f)
      if(isFromJar && source.getName == jarName) Some(f -> path) else None
    }
    Right(includedFiles)
  }
}

mergeStrategy in assembly := {
    case PathList("path", "to", "your", "DuplicatedClass.class") => includeFromJar("jarname.jar")
    case x => (mergeStrategy in assembly).value(x)
}

Tomtom