Scala 2 Append A Method To Class Body (Metaprogramming)

Question

I have been stuck on this issue for a week and don't seem to be getting anywhere. I am trying to copy some methods and fields from one class to another.

I have two phases that are involved in this. The first phase scans the code, finds the method defs that need to copied, and save the corresponding Tree

The second phase inserts this tree where needs to go. In order to simplify this question, let's forget about the copying and say that I am trying to insert a simple method def hello(): String = "hello" to the body of some class

The plugin runs after the typer (because I need the package information), and I am having a problem with injecting the type information properly. This results in an assertion exception in the later type checking stage (Full stacktrace at the bottom)

I asked about this in the metaprogramming discord and was pointed to the following resources.

Scala compiler plugin to rewrite method calls

https://contributors.scala-lang.org/t/scala-compiler-plugin-naming-issues-after-typer/2835

But neither yielded successful results unfortunately. I am assuming I have to take special care because the return type is a primitive (?), as the type gets interfaced through Predef

First Attempt:

Results in the error at the very end

tree match {
    case pl @ ClassDef(mods, name, tparams, e @ Template(parent, self, body)) =>
      parent.lift(1) match {
        case Some(a @ TypeTree()) =>
          a.original match {
            case AppliedTypeTree(Select(This(TypeName(s)), tpt), args) =>
              if (tpt.toString == "Policy") {
                val insert = q""" def q(): String = {"hello"}""".asInstanceOf[DefDef]
                val DefDef(dmodifiers, dname, dtparams, dvparams, dtpt, drhs) = insert
                val source = treeCopy.DefDef(insert, dmodifiers, dname, dtparams, dvparams, dtpt, drhs)

                val finalCopy = pl.copy(
                  mods,
                  name,
                  tparams,
                  Template(
                    parent,
                    self,
                    body.:+(
                      source
                    )
                  )
                )
                localTyper.typed(finalCopy)
              } else {
                super.transform(tree)
              }
            case _ => super.transform(tree)
          }
        case _ => super.transform(tree)
      }
      case _ => super.transform(tree)
    }

Instead of building the source, I have also tried manually constructing various things.

DefDef(
         Modifiers(),
         TermName("q"),
         List(),
         List(List()),
         TypeTree().setOriginal(Select(Select(Ident(scala), scala.Predef), TypeName("String"))), //attempt1
         Ident(TypeName("String")), //attemp2 
         TypeTree().setOriginal(Ident(TypeName("String"))), //attempt3
         gen.mkAttributedRef(typeOf[String].typeSymbol), //attempt 4
         Literal(Constant("hello")))

All resulting in the same error. Note that in the error, the class being printed have the method inserted but the type checker can not make sense of it for some reason

Following the suggestion on the contributors forum, I tried to set the ownership

val source = ... same as above
pl.symbol.owner.info.decls.unlink(pl.symbol)
localTyper.namer.enterDefDef(source)
source.symbol.owner.info.decls.enter(pl.symbol)

val finalCopy = pl.copy(....) //same as above

localTyper.namer.enterClassDef(finalCopy)
finalCopy.symbol.owner.info.decls.enter(finalCopy.symbol)
localTyper.typed(finalCopy)

But this completely screwed up everything and the compiler messed up the symbols and telling me fully implemented classes didn't implement the abstract members thus needed to be declared abstract

I have been going around in circles on this so if anybody have an idea what the best way to append a method to class body after the typer or have somewhat related examples, I would certainly appreciate it

 Exception in thread "main" java.lang.AssertionError: assertion failed: 
[error]   class UserPolicy extends AnyRef with prv.Main.Policy[prv.Main.User] {
[error]   <paramaccessor> private[this] val u: prv.Main.User = _;
[error]   def <init>(u: prv.Main.User): prv.Main.UserPolicy = {
[error]     UserPolicy.super.<init>();
[error]     ()
[error]   };
[error]   private[this] val data: prv.Main.User = UserPolicy.this.u;
[error]   <stable> <accessor> def data: prv.Main.User = UserPolicy.this.data;
[error]   protected def checkDeclassify(): prv.Main.User = {
[error]     def checkExpanded(): prv.Main.User = UserPolicy.this.data;
[error]     checkExpanded()
[error]   };
[error]   def unsafeUnwrap(reason: String): prv.Main.User = UserPolicy.this.data;
[error]   def q2(): String = "hello";
[error]   def q(): String = "hello"
[error] }
[error]      while compiling: <test>
[error]         during phase: method-wiring-phase
[error]      library version: version 2.13.1
[error]     compiler version: version 2.13.1
[error]   reconstructed args: -usejavacp
[error]   last tree to typer: type UserPolicy
[error]        tree position: <unknown>
[error]               symbol: <none>
[error]    symbol definition: <none> (a NoSymbol)
[error]       symbol package: <none>
[error]        symbol owners: 
[error]            call site: <none> in <none>
[error] == Source file context for tree position ==
[error]         at scala.reflect.internal.SymbolTable.throwAssertionError(SymbolTable.scala:170)
[error]         at scala.tools.nsc.typechecker.Typers$Typer.typedClassDef(Typers.scala:1876)
[error]         at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5794)
[error]         at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5886)
[error]         at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5948)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:254)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:195)
[error]         at scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2614)
[error]         at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2612)
[error]         at scala.reflect.internal.Trees$Template.transform(Trees.scala:517)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.$anonfun$transform$1(TypingTransformers.scala:47)
[error]         at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2625)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:37)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:32)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:333)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:195)
[error]         at scala.reflect.api.Trees$Transformer.transformTemplate(Trees.scala:2587)
[error]         at scala.reflect.internal.Trees$ModuleDef.$anonfun$transform$3(Trees.scala:370)
[error]         at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2625)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:37)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:32)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:24)
[error]         at scala.reflect.internal.Trees$ModuleDef.transform(Trees.scala:369)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:51)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:333)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:195)
[error]         at scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2614)
[error]         at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2612)
[error]         at scala.reflect.internal.Trees$PackageDef.$anonfun$transform$1(Trees.scala:316)
[error]         at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2625)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:37)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:32)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:24)
[error]         at scala.reflect.internal.Trees$PackageDef.transform(Trees.scala:316)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.$anonfun$transform$2(TypingTransformers.scala:49)
[error]         at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:2625)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:37)
[error]         at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:32)
[error]         at privacy.MethodWiring$MethodWiringPhase.transform(MethodWire.scala:333)
[error]         at privacy.MethodWiring$$anon$3.apply(MethodWire.scala:192)
[error]         at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:452)
[error]         at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:397)
[error]         at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1506)
[error]         at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1490)
[error]         at scala.tools.nsc.Global$Run.compileSources(Global.scala:1482)
[error]         at privacy.AnnotationFinderTest$.delayedEndpoint$privacy$AnnotationFinderTest$1(Test.scala:114)
[error]         at privacy.AnnotationFinderTest$delayedInit$body.apply(Test.scala:13)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1(App.scala:73)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:73)
[error]         at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
[error]         at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:921)
[error]         at scala.App.main(App.scala:73)
[error]         at scala.App.main$(App.scala:71)
[error]         at privacy.AnnotationFinderTest$.main(Test.scala:13)
[error]         at privacy.AnnotationFinderTest.main(Test.scala)

Planospore

Just in case, start of the discussion at Discord discord.com/channels/632150470000902164/875868146949554207/… – Brought

not sure, but perhaps you could glean something from the various mkMethod methods in github.com/wheaties/TwoTails/blob/master/core/src/main/scala/… – Cribbs

@SethTisue Thank for the suggestion. I did pick up some neat tricks from that code but unfortunately I still haven't figured out how to append which i think essentially is the problem. Notice that the plugin you sent have the advantage of rewriting existing methods – Planospore

I think as a next step, it makes sense to put an extra method in the ClassDef with a dummy annotation and see if it can be transformed. That would really help me nail down if it's the transformation or the appending – Planospore

Answer 1

Posting an answer so the question can be closed. It took me a while but I think I figured it out.

Thanks to @SethTisue for pointing me to TwoTails. I was able to correctly synthesize a method using the part of the code in that repo. However the bottom line is doing something like this after the typer is not trivially possible. Here is the reason why:

Say you are trying to synthesize and append a method m to a class C after the typer. The problem is if you are synthesizing this method, you will eventually invoke it somewhere new C().m. The membership is resolved during the typer, so the typer will never complete and throw an error method m is not a member of C. So,

1. If you don't require the package information to achieve this, you should do this after the parser

2. If you need the package information, this gets very tricky. You need to add a few new phases after the parser. I will omit the code because it is very lengthy but here is the gist of it.

   1. Phase 1: Accumulate the list of Class Names you will be appending to and their existing method, skip if it's a pre-known class

   2. Phase 2: Go through the code and accumulate a list of all the symbols that correspond to an instance of this class. ValDef and any parameters to the DefDef. If you have implemented Hindley Milner you will immediately identify the problem that will a way to distinguish similarly named symbols in different scopes. There is a lot of existing literature on this that you can read, I am skipping the details.

   3. Phase 3: Go through the code and accumulate a list of method names that are invoked on C but doesn't yet exist. You need to memorize the parameters and their types as well. Whether you need the return type or not really depends on what you are doing and/or if you want extra soundness/verification in a later step. You can skip this phase if the method you are appending is static and you already know what members will be missing in advance.

   4. Phase 4: Go through the code one last time and append a null method into C that with the proper name and types. Retuning null isn't the best thing, not sure if there is a better alternative.

   5. Later in the typer replace the appended method body with the proper one (the one you are copying)

The actual synthesis looks like this but as I mentioned above, if you actually want this to work, you will need to figure out all the stuff above.

    override def transform(tree: Tree): Tree = {
       val classesOfInterest = policyTypes.map(a => s"${a.packageName}.${a.typeName}").toList

       tree match {
         case pl @ ClassDef(mods, name, tparams, e @ Template(parent, self, body)) =>
           parent.lift(1) match {
            case Some(a @ TypeTree()) =>
              val original = a.original
              original match {
                case AppliedTypeTree(Select(This(TypeName(s)), tpt), args)=>
                  if (tpt.toString == "Policy") {
                    val insert                                                 = q"""... method to insert""".asInstanceOf[DefDef]
                    val DefDef(dmodifiers, dname, dtparams, dvparams, dtpt, drhs) = insert
                    val source                                                = treeCopy.DefDef(insert, dmodifiers, dname, dtparams, dvparams, dtpt, drhs)
                    //borrow the symbol of another method from the body. This is guaranteed because members like toString will be generated at this point
                    val xyz = mkNewMethodSymbol(body(body.length).symbol, TermName("q"))
                    localTyper.typedPos(tree.pos)(
                      treeCopy.ClassDef(
                        tree,
                        mods,
                        name,
                        tparams,
                        treeCopy.Template(
                          e,
                          e.parents,
                          e.self,
                          e.body :+ localTyper.typed(DefDef(xyz, mkNewMethodRhs(xyz, insert)))
                        )
                      )
                     )
                    )
                  } else {
                    super.transform(tree)
                  }
                case _ => super.transform(tree)
              }
            case _ => super.transform(tree)
          }
        case _ => super.transform(tree)
      }
    }

    def mkNewMethodSymbol(symbol: Symbol, name: TermName): Symbol = {
      val flags   = METHOD 
      val methSym = symbol.cloneSymbol(symbol.owner, flags, name)
      val param   = methSym.newSyntheticValueParam(definitions.IntTpe, TermName("indx"))

      methSym.modifyInfo {
        case GenPolyType(tparams, MethodType(params, res)) => GenPolyType(tparams, MethodType(params, res))
      }
      localTyper.namer.enterInScope(methSym)
    }

Planospore