SBT - Multi project merge strategy and build sbt structure when using assembly

Question

I have a project that consists of multiple smaller projects, some with dependencies upon each other, for example, there is a utility project that depends upon commons project. Other projects may or may not depend upon utilities or commons or neither of them.

In the build.sbt I have the assembly merge strategy at the end of the file, along with the tests in assembly being {}.

My question is: is this correct, should each project have its own merge strategy and if so, will the others that depend on it inherit this strategy from them? Having the merge strategy contained within all of the project definitions seems clunky and would mean a lot of repeated code.

This question applied to the tests as well, should each project have the line for whether tests should be carried out or not, or will that also be inherited?

Thanks in advance. If anyone knows of a link to a sensible (relatively complex) example that'd also be great.

Pardo

Answer 1

In my day job I currently work on a large multi-project. Unfortunately its closed source so I can't share specifics, but I can share some guidance.

1. Create a rootSettings used only by the root/container project, since it usually isn't part of an assembly or publish step. It would contain something like:

   lazy val rootSettings := Seq(
     publishArtifact := false,
     publishArtifact in Test := false
   )
   

2. Create a commonSettings shared by all the subprojects. Place the base/shared assembly settings here:

   lazy val commonSettings := Seq(
   
     // We use a common directory for all of the artifacts
     assemblyOutputPath in assembly := baseDirectory.value /
       "assembly" / (name.value + "-" + version.value + ".jar"),
   
     // This is really a one-time, global setting if all projects
     // use the same folder, but should be here if modified for
     // per-project paths.
     cleanFiles <+= baseDirectory { base => base / "assembly" },
   
     test in assembly := {},
   
     assemblyMergeStrategy in assembly := {
       case "BUILD" => MergeStrategy.discard
       case "logback.xml" => MergeStrategy.first
       case other: Any => MergeStrategy.defaultMergeStrategy(other)
     },
   
     assemblyExcludedJars in assembly := {
       val cp = (fullClasspath in assembly).value
       cp filter { _.data.getName.matches(".*finatra-scalap-compiler-deps.*") }
     }
   )
   

3. Each subproject uses commonSettings, and applies project-specific overrides:

   lazy val fubar = project.in(file("fubar-folder-name"))
     .settings(commonSettings: _*)
     .settings(
       // Project-specific settings here.
       assemblyMergeStrategy in assembly := {
         // The fubar-specific strategy
         case "fubar.txt" => MergeStrategy.discard
         case other: Any =>
           // Apply inherited "common" strategy
           val oldStrategy = (assemblyMergeStrategy in assembly).value
           oldStrategy(other)
       }
     )
     .dependsOn(
       yourCoreProject,
       // ...
     )
   

4. And BTW, if using IntelliJ. don't name your root project variable root, as this is what appears as the project name in the recent projects menu.

   lazy val myProjectRoot = project.in(file("."))
     .settings(rootSettings: _*)
     .settings(
       // ...
     )
     .dependsOn(
       // ...
     )
     .aggregate(
       fubar,
       // ...
     )
   

You may also need to add a custom strategy for combining reference.conf files (for the Typesafe Config library):

val reverseConcat: sbtassembly.MergeStrategy = new sbtassembly.MergeStrategy {
  val name = "reverseConcat"
  def apply(tempDir: File, path: String, files: Seq[File]): Either[String, Seq[(File, String)]] =
    MergeStrategy.concat(tempDir, path, files.reverse)
}

assemblyMergeStrategy in assembly := {
  case "reference.conf" => reverseConcat
  case other => MergeStrategy.defaultMergeStrategy(other)
}

Taranto

Upvoted. BTW, for first and last, how do I know which one comes first, and is the order guaranteed? – Aerotherapeutics

@AbhijitSarkar The order I use in the build.sbt file is to define the root and common settings blocks, then the subprojects and finally the root to tie them all together. The build order relies on the dependsOn and aggregate lists. – Taranto

@Taranto how about dependencies? A recent problem I faced was in an Akka project multiple third-party libraries contained a file reference.conf and so did I. The desired order was to concat the files from the jars first, followed by mine in the end. But mine ended up at the beginning of the merged file – Aerotherapeutics

@AbhijitSarkar Updated my answer with a custom strategy for that. – Taranto

Thanks. I think you're making the assumption that my file will be the first in the Seq, is that right? – Aerotherapeutics

Yes, it implicitly depends on the SBT-generated class path. – Taranto