sbt: dynamic aggregation of subproject

Question

I would like to invent a system to dynamically discover subprojects and aggregate them into my project automatically. Or at least configure this somehow.

To do this, I'm planning to have either a "modules" folder or an optional configuration file containing paths to the modules.

In any case I'd need to loop through subfolders (or loop through a list of paths from a configuration file), and aggregate each subproject. I don't know how to do that.

Currently I'm building in the Play framework with the build.sbt file. I would need to add the loop like this:

name := "mysite"
version := "1.0"
scalaVersion := "2.11.1"
lazy val root = (project in file(".")).enablePlugins(PlayJava)
//pseudocode:
foreach( folder in the 'modules' folder) { 
  lazy val module = (project in file(folder)).enablePlugins(PlayJava)
  root = root.dependsOn(module).aggregate(module)
}

Is there a way to do this?

EDIT 3: The code here is almost working:

object MyBuild extends Build {
  name := "mysite"
  version := "1.0"
  scalaVersion := "2.11.6"

  var m = new File("modules")
  var list = Seq[ProjectReference]()
  var deps = Seq[ClasspathDependency]()
  if (m.exists) {
    val subs = m.listFiles.filter ( _.isDirectory ).foreach { folder =>
      var modulePath = new File("modules", folder.getName)
      println("Found module " + modulePath)
      lazy val module:ProjectRef = ProjectRef(modulePath,folder.getName)
      lazy val dep:ClasspathDependency = ClasspathDependency(module, None)
      list = list :+ module
      deps = deps :+ dep
    }
  }

  lazy val root = Project(id = "mysite", base = file(".")).enablePlugins(PlayJava).aggregate(list:_*).dependsOn(deps:_*)
}

Edit 4:

See Dale Wijnand's solution below.

About the error: RuntimeException: No project 'myModule' in 'file:/Users/me/mysite/modules/myModule'. I fixed this using the solution from https://stackoverflow.com/a/28820578

Celebrate

What is the output of println("Found module " + folder.getName)? – Bestialize

it's "Found module foo" (my module folder is named "foo"). It seems that r.depends(module) fails. It may not be a Project object. For some reason. – Celebrate

Answer 1

Here:

project/Build.scala

import sbt._
import sbt.Keys._

import play.sbt._
import play.sbt.Play.autoImport._

object Build extends Build {
  val commonSettings: Seq[Setting[_]] = Seq(
    scalaVersion := "2.11.7"
  )

  lazy val modules = (file("modules") * DirectoryFilter).get.map { dir =>
    Project(dir.getName, dir).enablePlugins(PlayJava).settings(commonSettings: _*)
  }

  lazy val root = (project in file("."))
    .enablePlugins(PlayJava)
    .settings(
      name := "mysite",
      version := "1.0"
    )
    .settings(commonSettings: _*)
    .dependsOn(modules map (m => m: ClasspathDependency): _*)
    .aggregate(modules map (m => m: ProjectReference): _*)

  override lazy val projects = root +: modules
}

Note, make sure that the module directories don't also contain build.sbt files defining them as projects as that will cause confusing RuntimeException: No project 'x' in 'file:/x' type exception, see Can't use sbt 0.13.7 with Play subprojects

Coterie

That's the most complete solution so far. But I'm still getting the same error: RuntimeException: No project 'myModule' in 'file:/Users/me/mysite/modules/myModule'. Any idea why that would be? – Celebrate

Note: the subproject itself can be compiled fine on its own. – Celebrate

I've got it! The solution is here: stackoverflow.com/a/28820578 Basically the project root definition is duplicate in the main build and sub-project builds. The solution is to remove the root project definition in build.sbt for sub-projects. //lazy val root = ... – Celebrate

Indeed I was typing about that and asking you if you could check. I'll add it as a NB to the answer. Would that be a complete solution for you then? – Coterie

A related question, if you're interested: #31120940 – Celebrate

Answer 2

You can try something like this:

import sbt._
import sbt.Keys._

import play.sbt._
import play.sbt.Play.autoImport._

object Build extends Build {

  lazy val modules = (file("modules") * DirectoryFilter).get.map { dir =>
    Project(dir.getName, dir).enablePlugins(PlayJava)
  }

  lazy val root = (project in file("."))
    .enablePlugins(PlayJava)
    .settings(
      name := "mysite",
      version := "1.0"
    )
    .dependsOn(modules map (m => m: ClasspathDependency): _*)
    .aggregate(modules map (m => m: ProjectReference): _*)

  override lazy val projects = root +: modules
}

Note: Inspired by Dale's answer, but I had to remove the "commonSettings", otherwise it would not work for me.

Bestialize

I tried something similar. But root=root.dependsOn() doesn't work; that val is immutable. I get a "reassignment to val" error. I'm a newbie to Scala so please hold my hand with this. – Celebrate

The solution is to make the root a "var" instead of "val". Now I've got something almost working but I get a different issue. See my updated code above. – Celebrate

I'm sorry; I'm lost in this code. I don't know what the "t" and the "r" are. – Celebrate

I have a solution that works on Windows but not on Mac. It's all about folders at this point. I'll have it figured out soon. – Celebrate

OK I've updated my solution (see above) and I've narrowed down the problem to file() returning the wrong folder. It returns the root folder of the project and NOT the module folder. getCanonicalPath returns /Users/me/mysite/ instead of /Users/me/mysite/modules/myModule. This is very weird. – Celebrate

Is the code you posted above int he build.sbt-file? – Bestialize

I'm getting: java.lang.RuntimeException: No project 'test1' in 'file:/home/ubuntu/workspace/'. which suggests, that there is no project definition here for project "test1". It's probably looking for a lazy val which does not exist. – Bestialize

No it's the file project/Build.scala. I'm getting that error too, but printing the file's canonical path tells me that the sub-modules' Project uses the wrong path somehow. – Celebrate

Let us continue this discussion in chat. – Celebrate

I came up with another possible solution above. This time the folders are 100% correct. But it still refuses to load the subproject. I'm really losing patience. – Celebrate

@Bestialize You're missing a .get after filter to get a Seq[File] and then you can drop the file in file(folder) as it's already a file. – Coterie

Btw commonSettings probably didn't work for you as its using the improved 0.13.8 syntax. – Coterie

Yes. I use 0.13.5 and 0.13.7 depending on the stuff I work with. – Bestialize