How to add "provided" dependencies back to run/test tasks' classpath?

Question

Here's an example build.sbt:

import AssemblyKeys._

assemblySettings

buildInfoSettings

net.virtualvoid.sbt.graph.Plugin.graphSettings

name := "scala-app-template"

version := "0.1"

scalaVersion := "2.9.3"

val FunnyRuntime = config("funnyruntime") extend(Compile)

libraryDependencies += "org.spark-project" %% "spark-core" % "0.7.3" % "provided"

sourceGenerators in Compile <+= buildInfo

buildInfoPackage := "com.psnively"

buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, target)

assembleArtifact in packageScala := false

val root = project.in(file(".")).
  configs(FunnyRuntime).
  settings(inConfig(FunnyRuntime)(Classpaths.configSettings ++ baseAssemblySettings ++ Seq(
    libraryDependencies += "org.spark-project" %% "spark-core" % "0.7.3" % "funnyruntime"
  )): _*)

The goal is to have spark-core "provided" so it and its dependencies are not included in the assembly artifact, but to reinclude them on the runtime classpath for the run- and test-related tasks.

It seems that using a custom scope will ultimately be helpful, but I'm stymied on how to actually cause the default/global run/test tasks to use the custom libraryDependencies and hopefully override the default. I've tried things including:

(run in Global) := (run in FunnyRuntime)

and the like to no avail.

To summarize: this feels essentially a generalization of the web case, where the servlet-api is in "provided" scope, and run/test tasks generally fork a servlet container that really does provide the servlet-api to the running code. The only difference here is that I'm not forking off a separate JVM/environment; I just want to manually augment those tasks' classpaths, effectively "undoing" the "provided" scope, but in a way that continues to exclude the dependency from the assembly artifact.

Expand

I don't need to "add back" the 'provided' dependencies for running with sbt run or intellij. Check this out: github.com/dportabella/spark-examples – Skipbomb

Answer 1

For a similar case I used in assembly.sbt:

run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)) 

and now the 'run' task uses all the libraries, including the ones marked with "provided". No further change was necessary.

Update:

@rob solution seems to be the only one working on latest SBT version, just add to settings in build.sbt:

run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated,
runMain in Compile := Defaults.runMainTask(fullClasspath in Compile, runner in(Compile, run)).evaluated

Deprecate

@Deprecate Can confirm doesn't work with latest IntelliJ. My workaround was to just create a SBT configuration that i use to run & debug my local spark app, and call run-main explicitly like so: "run-main MySparkDriverMainClass" – Hukill

I don't need to "add back" the 'provided' dependencies for running with sbt run or intellij. Check this out: github.com/dportabella/spark-examples – Skipbomb

@DavidPortabella could you clarify what did you do to make it work? I'm struggling with the same problem. – Seisin

check this: github.com/typesafehub/mesos-spark-integration-tests/blob/… – Skipbomb

Works just fine in IntelliJ Ultimate 2016.2 for me... dropped this at the bottom of my build.sbt and used and SBT Task run configuration that does a run. – Postpaid

A solution avoiding use of <<= which is now deprecated in sbt is fullClasspath in Runtime := (fullClasspath in Compile).value – Particolored

So, after struggling with this as well, I'll hopefully be able to clear up @GauravShah 's somewhat worrying comment: This (and in particular Jason Fonseca's version) will NOT work for the standard "run application" IntelliJ Run-Configuration. Instead you have to create a corresponding SBT-Task Configuration ("run") and run that. At the same time, I consider this a bug of IntelliJ, which should, when the sbt plugin is installed, use the run-Task for "run application", and respect build.sbt's configuration. – Justus

The configuration that worked for me with this is run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated and runMain in Compile := Defaults.runMainTask(fullClasspath in Compile, runner in(Compile, run)).evaluated (in SBT version 0.13.13) – Anthropoid

@RickMoritz so any way to fix it for "run application" – Hawaii

In case anyone wants the previous working version by @Anthropoid in slash syntax - working with sbt 1.4.7 Compile / run := Defaults.runTask(Compile / fullClasspath, Compile / run / mainClass, Compile / run /runner).evaluated, Compile / runMain := Defaults.runMainTask(Compile / fullClasspath, Compile / run / runner).evaluated – Helmick

Answer 2

Adding to @douglaz' answer,

runMain in Compile <<= Defaults.runMainTask(fullClasspath in Compile, runner in (Compile, run))

is the corresponding fix for the runMain task.

Metallurgy

Does anyone know how the corresponding line for sbt-start-script has to look like? – Metallurgy

Any idea why IntelliJ doesn't respect this? – Hukill

Actually IntelliJ will automatically tick the include dependencies with "Provided" scope box in the Run/Debug configuration dialog, when you reload your project after having used the solutions from the answers on this page. At least in version 2019.2 of IntelliJ. Which is very nice! – Reimpression

Answer 3

Another option is to create separate sbt projects for assembly vs run/test. This allows you to run sbt assemblyProj/assembly to build a fat jar for deploying with spark-submit, as well as sbt runTestProj/run for running directly via sbt with Spark embedded. As added benefits, runTestProj will work without modification in IntelliJ, and a separate main class can be defined for each project in order to e.g. specify the spark master in code when running with sbt.

val sparkDep = "org.apache.spark" %% "spark-core" % sparkVersion

val commonSettings = Seq(
  name := "Project",
  libraryDependencies ++= Seq(...) // Common deps
)

// Project for running via spark-submit
lazy val assemblyProj = (project in file("proj-dir"))
  .settings(
    commonSettings,
    assembly / mainClass := Some("com.example.Main"),
    libraryDependencies += sparkDep % "provided"
  )

// Project for running via sbt with embedded spark
lazy val runTestProj = (project in file("proj-dir"))
  .settings(
    // Projects' target dirs can't overlap
    target := target.value.toPath.resolveSibling("target-runtest").toFile,
    commonSettings,
    // If separate main file needed, e.g. for specifying spark master in code
    Compile / run / mainClass := Some("com.example.RunMain"),
    libraryDependencies += sparkDep
  )

Lumberjack

Answer 4

If you use sbt-revolver plugin, here is a solution for its "reStart" task:

fullClasspath in Revolver.reStart <<= fullClasspath in Compile

UPD: for sbt-1.0 you may use the new assignment form:

fullClasspath in Revolver.reStart := (fullClasspath in Compile).value

Oxy

When I tried this I got [error] /Users/irashid/github/pub/spark_streaming_demo/project/Build.scala:69: value fullClasspath is not a member of sbt.Def.Initialize[sbt.InputTask[Unit]] [error] possible cause: maybe a semicolon is missing before `value fullClasspath'? [error] fullClasspath in reStart <<= fullClasspath in Compile [error] ^ [error] one error found [error] (compile:compile) Compilation failed. any suggestions? I'm probably missing something basic, but trying reStart.fullClasspath, or reStart::fullClasspath or Revolver.reStart::fullClasspath all give some error – Morrell

I've tried it right now, it works for me. But I use the *.sbt file, not the deprecated Build.sbt. IDK. Maybe some import is missing? – Oxy