Splicing a passed function body into a macro-rewritten expression

Question

I was playing with Scala 2.11's new macro features. I wanted to see if I could do the following rewrite:

forRange(0 to 10) { i => println(i) }

// into

val iter = (0 to 10).iterator
while (iter.hasNext) {
  val i = iter.next
  println(i)
}

I think I got fairly close with this macro:

def _forRange[A](c: BlackboxContext)(range: c.Expr[Range])(func: c.Expr[Int => A]): c.Expr[Unit] = {
  import c.universe._

  val tree = func.tree match {
    case q"($i: $t) => $body" => q"""
        val iter = ${range}.iterator
        while (iter.hasNext) {
          val $i = iter.next
          $body
        }
      """
    case _ => q""
  }

  c.Expr(tree)
}

This produces the following output when called as forRange(0 to 10) { i => println(i) } (at least, it's what the show function gives me on the resultant tree):

{
  val iter = scala.this.Predef.intWrapper(0).to(10).iterator;
  while$1(){
    if (iter.hasNext)
      {
        {
          val i = iter.next;
          scala.this.Predef.println(i)
        };
        while$1()
      }
    else
      ()
  }
}

That looks like it should work, but there's a conflict between my manually defined val i and the i referenced in the spliced-in function body. I get the following error:

ReplGlobal.abort: symbol value i does not exist in$line38.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw. error: symbol value i does not exist in scala.reflect.internal.FatalError: symbol value i does not exist in $line38.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.

And then a rather large stack trace, resulting in an "Abandoned crashed session" notification.

I can't tell if this is a problem with my logic (you simply can't splice in a function body that references a closed-over variable), or if it's a bug with the new implementation. The error reporting certainly could be better. It may be exacerbated by the fact that I'm running this on the Repl.

Is it possible to pull apart a function, separating the body from the closed-over terms, and rewrite it in order to splice the logic directly into a resulting tree?

Archeozoic

@KChalous Did you ever manage to fix this now that resetAllAttrs has been removed from Scala 2.11? I have the exact same problem, and I am desperate to fix it! – Redman

@AndrewBate According to online documentation, there's a resetLocalAttrs that still exists which should cover most cases. Dunno if this is one of them, but it's worth a shot. Reference: docs.scala-lang.org/overviews/macros/changelog211.html – Archeozoic

@AndrewBate and according to the scalamacros project on github, resetLocalAttrs has been renamed to untypecheck. Reference: github.com/scalamacros/resetallattrs – Archeozoic

@KChalous I've investigated a little more, and in this exact example, I think that untypecheck is enough. However, when I try to match the pattern q"for ($i <- $collection) $body", I need the old resetAllAttrs (or so it seems). I've used the resetAllAttrs library for Scala 2.11 macros, and it works, whereas untypecheck alone does not, but I quickly run into the usual problem with resetAllAttrs corrupting parents trees. – Redman

@AndrewBate I'm definitely no expert on this, unfortunately. It looks like, via that github link, you can get resetAllAttrs back, but they removed it specifically because of the tree-corrupting issue you mentioned. Still, if you need it, look into github.com/scalamacros/resetallattrs – Archeozoic

Thanks, I've tried that, and it does corrupt my trees in anything other than a trivial example. I'll keep looking for a solution... – Redman

Answer 1

When in doubt, resetAllAttrs:

import scala.language.experimental.macros
import scala.reflect.macros.BlackboxContext

def _forRange[A](c: BlackboxContext)(range: c.Expr[Range])(
  func: c.Expr[Int => A]
): c.Expr[Unit] = {
  import c.universe._

  val tree = func.tree match {
    case q"($i: $t) => $body" => q"""
        val iter = ${range}.iterator
        while (iter.hasNext) {
          val $i = iter.next
          ${c.resetAllAttrs(body)} // The only line I've changed.
        }
      """
    case _ => q""
  }

  c.Expr(tree)
}

And then:

scala> def forRange[A](range: Range)(func: Int => A) = macro _forRange[A]
defined term macro forRange: [A](range: Range)(func: Int => A)Unit

scala> forRange(0 to 10) { i => println(i) }
0
1
2
3
4
5
6
7
8
9
10

In general, when you're grabbing a tree from one place and plopping it somewhere else, it's likely going to be necessary to use resetAllAttrs to get all the symbols right.

Titian

Wasn't even aware this existed! Very nice. – Archeozoic

Unfortunately it was removed in 2.11. I ran into the same issue and couldn't resolve it with resetLocalAttrs (deprecated) or untypecheck. – Dreadful

Answer 2

Oscar Boykin pointed out on Twitter that my previous answer no longer works, and it wasn't a very complete answer anyway—it addresses the problem pointed out by the OP on Scala 2.10, but it's not careful about hygiene—if you wrote iter => println(iter) you'd get a compile-time failure, for example.

A better implementation for 2.11 would use a Transformer to rewrite the tree after un-typechecking it:

import scala.language.experimental.macros
import scala.reflect.macros.blackbox.Context

def _forRange[A](c: Context)(r: c.Expr[Range])(f: c.Expr[Int => A]): c.Tree = {
  import c.universe._

  f.tree match {
    case q"($i: $_) => $body" =>
      val newName = TermName(c.freshName())
      val transformer = new Transformer {
        override def transform(tree: Tree): Tree = tree match {
          case Ident(`i`) => Ident(newName)
          case other => super.transform(other)
        }
      }

      q"""
        val iter = ${r.tree}.iterator
        while (iter.hasNext) {
          val $newName = iter.next
          ${ transformer.transform(c.untypecheck(body)) }
        }
      """
  }
}

def forRange[A](r: Range)(f: Int => A): Unit = macro _forRange[A]

Which works like this:

scala> forRange(0 to 10)((i: Int) => println(i))
0
1
2
3
4
5
6
7
8
9
10

Now it doesn't matter what variable name we use in our function literal, since it'll just be replaced with a fresh variable anyway.

Titian

Where can documentation/manual be found for extended macroses techniques that goes beyond simple text generation? Like untypecheck, context.info and other methods that works with code as scala objects with types classes and so on instead of text snippets. I've found docs.scala-lang.org/overviews/macros/usecases.html but it contains only techniques related to text substitution. – Geometer

@Geometer I don't have a good answer for that. I've got a bunch of demo projects and blog posts and SO answers, and a lot of other people do too, but they're not indexed in any way. – Titian