Scala - How to compile code from an external file at runtime?

Question

I want to design a Scala program that accepts Scala files as parameters which can customize the execution of the program. In particular, I want to supply at runtime files that contain implementations of methods that will be invoked by the program. How can I properly depend on external files and invoke their methods dynamically at runtime? Ideally, I would also like those files to be able to depend on methods and classes in my program.

Example Scenario: I have a function that contains the line val p: Plant = Greenhouse.getPlant(), and the Greenhouse class with the getPlant method is defined in one of the files that will be supplied at runtime. In that file, the method getPlant returns a Rose, where Rose <: Plant and Plant is defined in the original program. How do I achieve (or approximate) this interdependency, assuming the files are only joined at runtime and not at compile-time?

Rosenberry

https://mcmap.net/q/64843/-quot-eval-quot-in-scala – Dennison

Answer 1

Here's how to do it using only standard Scala. The non-obvious stuff is all in GreenhouseFactory:

package customizable

abstract class Plant

case class Rose() extends Plant

abstract class Greenhouse {
  def getPlant(): Plant
}

case class GreenhouseFactory(implFilename: String) {
  import reflect.runtime.currentMirror
  import tools.reflect.ToolBox
  val toolbox = currentMirror.mkToolBox()
  import toolbox.u._
  import io.Source

  val fileContents = Source.fromFile(implFilename).getLines.mkString("\n")
  val tree = toolbox.parse("import customizable._; " + fileContents)
  val compiledCode = toolbox.compile(tree)

  def make(): Greenhouse = compiledCode().asInstanceOf[Greenhouse]
}

object Main {
  def main(args: Array[String]) {
    val greenhouseFactory = GreenhouseFactory("external.scala")
    val greenhouse = greenhouseFactory.make()
    val p = greenhouse.getPlant()

    println(p)
  }
}

Put your override expression in external.scala:

new Greenhouse {
  override def getPlant() = new Rose()
}

The output is:

Rose()

The only tricky thing is that GreenhouseFactory needs to prepend that import statement to provide access to all the types and symbols needed by the external files. To make that easy, make a single package with all those things.

The compiler ToolBox is sort of documented here. The only thing you really need to know, other than the weird imports, is that toolbox.parse converts a string (Scala source code) into an abstract syntax tree, and toolbox.compile converts the abstract syntax tree into a function with signature () => Any. Since this is dynamically compiled code, you have to live with casting the Any to the type that you expect.

Histology

Very clear explanation, thanks. When you say that GreenhouseFactory needs to import all the types and symbols that will be accessed externally, you're referring to the import string that is prepended to fileContents before being passed to parse, right? Or are you saying that the GreenhouseFactory method itself actually needs to import everything? – Rosenberry

I mean the string prepended to fileContents. I'll edit that right now. – Histology

Ok. Do I need to be concerned with type checking the file or is that already handled in the Toolbox methods that are called? – Rosenberry

The ToolBox methods will throw a run-time exception if there are any errors, syntactical or otherwise. If the new in the external file doesn't produce an object that works with the cast in .make, that will cause a run-time exception, too. – Histology

Is it possible to compile a String of Java code in a similar way? – Aparri

@Aparri Have a look on this mike-java.blogspot.in/2008/03/java-6-compiler-api-tutorial.html. May be it can help you. :-) – Groomsman

Do not forget to add "org.scala-lang" % "scala-compiler" % scalaVersion.value dependency. – Discrete

Does it use a different class loader to get the class and instance? – Recite

Answer 2

Scala doesn't provide this sort of functionality natively. The easiest way I know of to do this is with the Twitter "util-eval" library. This library wraps the necessary calls to the Scala compiler and the various class loading rituals, saving you enormous amounts of effort. The calls sequence to do what you're describing would look something like

val eval = new Eval()
val greenhouse = eval.apply[Greenhouse](new File("path/to/MyGreenhouse.scala"))
val plant = greenhouse.getPlant()

Your dynamically loaded Scala file needs to contain an expression, not a class per se, but that's pretty easy to do, basically like this.

new Greenhouse{
    def getPlant() = //thing to return the plant 
}

As I understand it, Twitter uses (or at least used) this functionality to have it's configuration files as Scala rather than properties/json/xml.

Gavotte

looks like this is no longer supported github.com/twitter/util/issues/105 – Iliad