How to compile and execute scala code at run-time in Scala3?

Question

I would like to compile and execute Scala code given as a String at run-time using Scala3. Like for example in Scala 2 I would have used Reflection

import scala.reflect.runtime.universe as ru
import scala.tools.reflect.ToolBox
val scalaCode = q"""println("Hello world!")"""
val evalMirror = ru.runtimeMirror(this.getClass.getClassLoader)
val toolBox = evalMirror.mkToolBox()
toolBox.eval(scalaCode) //Hello world!

If I try to run this code in Scala3 I get

Scala 2 macro cannot be used in Dotty. See https://dotty.epfl.ch/docs/reference/dropped-features/macros.html
To turn this error into a warning, pass -Xignore-scala2-macros to the compiler

How can I translate this code in Scala3 ?

Blowtube

Have you read Scala 3 metaprogramming documentation? Did you try something? – Toffeenosed

@GaëlJ Yes I did, but it is for the most parts incomplete and many of the links are broken. I understood very little, that's why I am looking for help. Do you know how to do it ? – Blowtube

Why would you do that? Runtime reflection should only be used in case there is no other solution, which is quite quite rare – Maurits

Please report broken links to github.com/scala/docs.scala-lang/issues – Cretonne

What we are saying is that you should give us more context on the why you want to do that. Depending on the reasons we will be able to give you ideas on how to approach your goal with Scala 3. – Toffeenosed

@Blowtube Actually, your code is very close to working. It's enough to replace q"..." (a Scala 2 macro) with toolbox.parse("..."). See update. – Iosep

Answer 1

Scala 2 version of this answer is here: How can I run generated code during script runtime?

In Scala 3:

• For example you can use Li Haoyi's Ammonite

ammonite.Main(verboseOutput = false).runCode("""println("Hello, World!")""")
// Hello, World!

build.sbt

scalaVersion := "3.1.3"
libraryDependencies += "com.lihaoyi" % "ammonite" % "2.5.4-22-4a9e6989" cross CrossVersion.full
excludeDependencies ++= Seq(
  ExclusionRule("com.lihaoyi", "sourcecode_2.13"),
  ExclusionRule("com.lihaoyi", "fansi_2.13"),
)

• Or you can try Eugene Yokota's Eval

com.eed3si9n.eval.Eval()
  .evalInfer("""println("Hello, World!")""")
  .getValue(this.getClass.getClassLoader)
// Hello, World!

build.sbt

scalaVersion := "3.2.0"
libraryDependencies += "com.eed3si9n.eval" % "eval" % "0.1.0" cross CrossVersion.full

• Or you can try my implementation

com.github.dmytromitin.eval.Eval[Unit]("""println("Hello, World!")""")
// Hello, World!

scalaVersion := "3.2.1"
libraryDependencies += "com.github.dmytromitin" %% "eval" % "0.1"

• Also you can use standard Scala 3 REPL interpreter

dotty.tools.repl.ScriptEngine().eval("""println("Hello, World!")""")
// Hello, World!

build.sbt

scalaVersion := "3.1.3"
libraryDependencies += scalaOrganization.value %% "scala3-compiler" % scalaVersion.value

• If you have a scala.quoted.Expr '{...} (a statically typed wrapper over an abstract syntax tree scala.quoted.Quotes#Tree) rather than plain string then you can use runtime multi-staging

import scala.quoted.*
given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
staging.run('{ println("Hello, World!") })
// Hello, World!

build.sbt

scalaVersion := "3.1.3"
libraryDependencies += scalaOrganization.value %% "scala3-staging" % scalaVersion.value

• All of the above is to run Scala 3 code in Scala 3. If we want to run Scala 2 code in Scala 3 then we can still use Scala 2 reflective Toolbox. Scala 2 macros don't work, so we can't do runtime.currentMirror or q"..." but can do universe.runtimeMirror or tb.parse

import scala.tools.reflect.ToolBox // implicit 

val tb = scala.reflect.runtime.universe
  .runtimeMirror(getClass.getClassLoader)
  .mkToolBox()
tb.eval(tb.parse("""println("Hello, World!")"""))
// Hello, World!

build.sbt

scalaVersion := "3.1.3"
libraryDependencies ++= scalaOrganization.value % "scala-compiler" % "2.13.8"

• Also to run Scala 2 code in Scala 3 you can use standard Scala 2 REPL interpreter

scala.tools.nsc.interpreter.shell.Scripted()
  .eval("""System.out.println("Hello, World!")""")
// Hello, World!

build.sbt

scalaVersion := "3.1.3"
libraryDependencies ++= scalaOrganization.value % "scala-compiler" % "2.13.8"

• Also you can use JSR223 scripting. Depending on whether you have scala3-compiler or scala-compiler in your classpath you will run Scala 3 or Scala 2 (one of the two above script engines: Scala 3 dotty.tools.repl.ScriptEngine or Scala 2 scala.tools.nsc.interpreter.shell.Scripted). If you have both the dependency added first wins.

new javax.script.ScriptEngineManager(getClass.getClassLoader)
  .getEngineByName("scala")
  .eval("""println("Hello, World!")""")
// Hello, World!

If you'd like to have a better control what dependency is used (without re-importing the project) you can use Coursier and specify class loader

import coursier.* // libraryDependencies += "io.get-coursier" %% "coursier" % "2.1.0-M6-53-gb4f448130" cross CrossVersion.for3Use2_13
val files = Fetch()
  .addDependencies(
    Dependency(Module(Organization("org.scala-lang"), ModuleName("scala3-compiler_3")), "3.2.0"),
    // Dependency(Module(Organization("org.scala-lang"), ModuleName("scala-compiler")), "2.13.9")
  )
  .run()

val classLoader = new java.net.URLClassLoader(
  files.map(_.toURI.toURL).toArray,
  /*getClass.getClassLoader*/null // ignoring current classpath
)
new javax.script.ScriptEngineManager(classLoader)
  .getEngineByName("scala")
  .eval("""
    type T = [A] =>> [B] =>> (A, B) // Scala 3
    //type T = List[Option[A]] forSome {type A} // Scala 2
    System.out.println("Hello, World!")
  """)
// Hello, World!

• You can implement Eval in Scala 3 yourself using actual compiler

import dotty.tools.io.AbstractFile
import dotty.tools.dotc.core.Contexts.Context
import dotty.tools.dotc.Driver
import dotty.tools.dotc.util.SourceFile
import dotty.tools.io.{VirtualDirectory, VirtualFile}
import java.net.URLClassLoader
import java.nio.charset.StandardCharsets
import dotty.tools.repl.AbstractFileClassLoader
import scala.io.Codec
import coursier.{Dependency, Module, Organization, ModuleName, Fetch}

  // we apply usejavacp=true instead
//  val files = Fetch()
//    .addDependencies(
//       Dependency(Module(Organization("org.scala-lang"), ModuleName("scala3-compiler_3")), "3.1.3"),
//    )
//    .run()
//
//  val depClassLoader = new URLClassLoader(
//    files.map(_.toURI.toURL).toArray,
//    /*getClass.getClassLoader*/ null // ignoring current classpath
//  )

val code =
  s"""
     |package mypackage
     |
     |object Main {
     |  def main(args: Array[String]): Unit = {
     |    println("Hello, World!")
     |  }
     |}""".stripMargin

val outputDirectory = VirtualDirectory("(memory)")
compileCode(code, List()/*files.map(f => AbstractFile.getFile(f.toURI.toURL.getPath)).toList*/, outputDirectory)
val classLoader = AbstractFileClassLoader(outputDirectory, this.getClass.getClassLoader/*depClassLoader*/)
runObjectMethod("mypackage.Main", classLoader, "main", Seq(classOf[Array[String]]), Array.empty[String])
// Hello, World!

def compileCode(
                 code: String,
                 classpathDirectories: List[AbstractFile],
                 outputDirectory: AbstractFile
               ): Unit = {
  class DriverImpl extends Driver {
    private val compileCtx0 = initCtx.fresh
    given Context = compileCtx0.fresh
      .setSetting(
        compileCtx0.settings.classpath,
        classpathDirectories.map(_.path).mkString(":")
      ).setSetting(
        compileCtx0.settings.usejavacp,
        true
      ).setSetting(
        compileCtx0.settings.outputDir,
        outputDirectory
      )
    val compiler = newCompiler
  }

  val driver = new DriverImpl
  import driver.given Context

  val sourceFile = SourceFile(VirtualFile("(inline)", code.getBytes(StandardCharsets.UTF_8)), Codec.UTF8)
  val run = driver.compiler.newRun
  run.compileSources(List(sourceFile))
  // val unit = run.units.head
  // println("untyped tree=" + unit.untpdTree)
  // println("typed tree=" + unit.tpdTree)
}

def runObjectMethod(
                     objectName: String,
                     classLoader: ClassLoader,
                     methodName: String,
                     paramClasses: Seq[Class[?]],
                     arguments: Any*
                   ): Any = {
  val clazz = Class.forName(s"$objectName$$", true, classLoader)
  val module = clazz.getField("MODULE$").get(null)
  val method = module.getClass.getMethod(methodName, paramClasses*)
  method.invoke(module, arguments*)
}

(previous version)

build.sbt

scalaVersion := "3.1.3"
libraryDependencies += scalaOrganization.value %% "scala3-compiler" % scalaVersion.value

See also: get annotations from class in scala 3 macros (hacking multi-staging programming in Scala 3 and implementing our own eval instead of Scala 2 context.eval or staging.run forbiden in Scala 3 macros).

• See also

An intro to the Scala Presentation Compiler

Parsing scala 3 code from a String into Scala 3 AST at runtime

Scala 3 Reflection

Help with dotty compiler and classloading at runtime

Iosep

github.com/propensive/fury/blob/main/src/core/compilation.scala – Iosep

Excellent, and thanks. But what if we want to create an Expr[T] from a String instead of just running it and receiving its calculated value? Is it possible to parse strings to typed expressions, for example load some file contents as a string, and then parse it and analyze it before running it? – Midrash

Have you found a way @Midrash ? – Washerman

Dear @Washerman according to my research it seems that Scala3 designers intentionally dropped this capability from meta programming design agenda (e.g. see contributors.scala-lang.org/t/… from Professor Odersky). Such design philosophy is justified by the fact that creating interpreters which only accepts special Scala3 code snippet string and disregard other valid snippets should be done per application requirements and hence is a developer task because it includes many per application design decisions. – Midrash

On the other side, being capable of creating expressions from any arbitrary valid Scala3 snippet string, reduces readability and degrades scalability. Anyway you can create an interpreter large enough to embrace whole Scala3 language constructs, if your application requirements dictates. Scala3 designers only provided infrastructures to systematically embed string interpreters of intended subset of Scala3 language constructs, since merely no real world application may need to embed a whole Scala3 language interpreter. – Midrash