How to do a sbt-(local)multi-project without the same project-root?

Question

I want to work on multiple sbt-projects at the same time in intellij. The projects are "one way dependent", meaning that one is the (reusable) core and the other is an actual application built upon that core. Both are currently in development.

I want the core to reside in another base/root-directory than the application, so:

- /core
-- build.sbt

- /application
-- build.sbt

I want to be able to

1. modify both projects in the same IntelliJ-window
2. leave both projects in their respective folder (no wrapper-folder around them!). Core will be used in other applications as well, applications that are no siblings to the current "application", so I do not want them to reside under the same root-folder!

What I've tried and which problems I've found so far:

setups like

lazy val core = project.in(file("../core"))
lazy val application = project.in(file(".")).dependsOn(core)

are not working, because an sbt asserts that the directory of each project in a multi-project-setup are contained in the same build root:

sbt java.lang.AssertionError: assertion failed: Directory /core is not contained in build root /application

setups like

lazy val core = RootProject(file("../core"))
lazy val application = project.in(file(".")).dependsOn(core)

are not a solution because:

1. I can not have both projects in one IntelliJ-window then
2. Strangely the classes of core are not found in application, although the imports worked right away

Now I am a sbt-newbie and I guess (and hope) that there must be a solution for this problem. I can't be the only one wanting to separate my projects without a wrapper-layer and still be able to modify them in the IDE of my choice.

edit:

@OlegRudenko´s solution is semi-working for me. As core has some dependencies as well, I cannot compile or work with it in application.

core pulls in some dependencies like e.g. a Logger and when I'm in application and try to use a component of core the compiler screams at me, because it can't find the dependencies in core (e.g. the logger).

Also, core pulls in e.g. lwjgl and I want to use some components of it in application, no chance, because it can't find the packages for that dependency of core.

For now what I do is a hacky non-solution. I just develop both core and application in the same intellij-project and keep the git-repo private.

This is not a solution at all, as I want to open source core while application is closed source for now and I still want to work on both at the same time, refine the core etc.

Disproportion

I would make inside your root project a symbolic link to the core, which will be kept at its original place. Can you do so? – Discovert

@OlegRudenko: Yes, but that would mean manually managing it on every machine I work one. As the project(s) will go open source one day it's kind of tedious to assume the same of every developer :( – Disproportion

the folder structure should be same on all machines. So, if, for example, SBT Build File will create the symbolic link automatically, then it could be a solution for your need. Right? If yes, I will investigate it to provide details as an answer. – Discovert

@OlegRudenko: Yes, if it is the same on every Operating System it would be fine by me. – Disproportion

@Teolha take a look at the project organization I use for my projects github.com/alexandrnikitin/netty-scala-template/tree/master/… – Deceit

@AlexandrNikitin What exactly should I see there? – Disproportion

This is one of the correct times to use git submodules. – Babb

@pfn: How is a git submodule helping with my (separated) directory-structure? – Disproportion

The point is you don't, you create a top-level project which is the outer git container, and git submodules for each core and app project. core and app don't know each other, other than app depending on core. The git trees are completely independent, which seems to be your overall goal. (being able to keep one open-source while the other is not) – Babb

Then again I would need an other container, which is not really what I'd prefer... but alas, I'll have a look. – Disproportion

Answer 1

If some framework does not allow me something, I prefer to not fight but to find a compromise.

SBT does not allow to have a sub-project outside of project root. So my solution would be to create a symbolic link inside my project to the sub-project, which is located outside:

common/
common/core/  - I want this as a sub-project inside of my project
application/
application/ui/  - This is my project

application/ui/core/ -> ../../common/core/  - This is my goal

It has to

• work automatically
• work on different operating systems
• be clear and understandable
• use standard tools

Solution is to add following lines into the SBT file:

val __createLinkToCore = {
  import java.nio.file.{FileSystemException, Files, Paths}
  import scala.util.Try

  val corePath = Paths.get("..", "..", "common", "core")
  if(!Files.exists(corePath)) throw new IllegalStateException("Target path does not exist")

  val linkPath = Paths.get("core")
  if(Files.exists(linkPath)) Files.delete(linkPath)

  Try {Files.createSymbolicLink(linkPath, corePath)}.recover {
    case cause: FileSystemException if System.getProperty("os.name").toLowerCase.contains("windows") =>
      val junctionCommand = Array("cmd", "/c", "mklink", "/J", linkPath.toAbsolutePath.normalize().toString, corePath.toAbsolutePath.normalize().toString)
      val junctionResult = new java.lang.ProcessBuilder(junctionCommand: _*).inheritIO().start().waitFor()
      if(junctionResult != 0) throw new Exception("mklink failed", cause)
      linkPath
  }.get
}

It is executed every time when SBT is started but before any action on your code - before compilation and so on. It does NOT recreate the link on every compilation - only once per SBT start.

The code does following:

• get path to desired folder (and fail if it doesn't exist)
• get path to link (and delete if it exists)
• try to create a new link using Java 7+ API
• if it runs on Windows 7+ and without admin right it will fail. In this case I create a junction using standard but windows-specific tool mklink /J

Limits of this solution:

• it will not work on FAT file systems
• have you seen other limits? let me know!

Discovert

Tested on IDEA 14 & Windows 7 – Discovert

Sadly this doesn't work quite right for me. I can add the core just fine and the application even finds the classes, but when I try to compile it (the application) it complains that in the core-classes imports can't be found. Like if I have a import tld.someclass within core.someotherclass and when application imports the core.someotherclass it complains that there it can't find import tld.someclass (although everything is fine if I just look at the core-package). Maybe you can check again and add the whole .sbt file? I've manually added an unmangedSourceDirectory, that may be the prob – Disproportion

Also, if you find a way to do it, can you check if it works if "core" has some (native) libraries as a dependency? I would need these as well in the "application". – Disproportion

Could you look into this again, please? – Disproportion

Sorry, I cannot review it now. But I hope to have more time in a month. – Discovert

Does this problem come in SBT or in IDEA only? Could it be related to youtrack.jetbrains.com/issue/SCL-7992? – Discovert

Answer 2

I don't think this is possible in SBT (but I'd love to hear otherwise).

Here's what I'm doing in IntelliJ to work around this issue:

• Open "application" in IntelliJ; refresh the SBT definition
• Turn off SBT auto-update (file -> settings -> build -> SBT)
• Do file -> new -> module from existing sources, choose the "core" checkout dir, import it from SBT
  • Tick “download sources”, but not “auto-update”
• You should now have both “application” and “core” in the same IntelliJ window (but "application" still depends on a "core" JAR not your live sources)
• Right click on the "application" project in IntelliJ and choose “open module settings”
  • Open the “dependencies” tab
  • Find the "core" SBT dependency(-ies) in the libraries and delete them
  • Add a "module dependency" on the "core" project
• You should manually rebuild the project at this point, as IntelliJ has been known to get confused about classpaths when adding in modules.
• You can now develop with the two projects as a single codebase in IntelliJ
  • Note that SBT command line tools like “sbt test” in "application" will continue to use the "core" JAR and not the locally modified sources
  • (The SBT Build.scala file in "application" still references the "core" JAR by version number)
• When you are ready to commit, you will need to:
  • Commit your changes to "core", update the version number
  • Run "sbt publishLocal" in "core", make a note of the version numbers generated
  • Update the Build.scala file in "application" to reference the new "core" version
  • Run "sbt test" or similar to check that everything works in SBT mode
  • Commit the "application" changes, push both

Earvin

Answer 3

You could use ProjectRef:

lazy val my_application = (project in file ("."))
    .aggregate(my_core)
    .dependsOn(my_core)

lazy val my_core = ProjectRef(file("../my_core"), "my_core")

Then, regarding the directory structure, my_application and my_core are siblings.

Brotherson