scala exception in for-comprehension with type annotation

Question

I am trying to understand what seems like strange behavior when dealing with nulls and type annotations inside a for-comprehension.

As an example:

def f(): String = null

for {
  a <- Option("hello")
  b = f()
} yield (a, b)

results in the expected:

//> res0: Option[(String, String)] = Some((hello,null)) 

however, if I add a type annotation to the type of b

def f(): String = null

for {
  a <- Option("hello")
  b: String = f()
} yield (a, b)

then I get a runtime exception:

//> scala.MatchError: (hello,null) (of class scala.Tuple2)

Why does this happen? Isn't b implicitly of type String in the first example anyway? What does the explicit type annotation in the second example change?

(Note, examples were run in Scala 2.11.4)

Dobbins

The second example compiles to a match, as you can see using reify in the repl: import scala.reflect.runtime.universe._; reify {for {... } }. I can't tell you why though. AIUI null doesn't match because matching works on the runtime type (or value), even though b has a compile-time type of String; this is in some sense a hole in the type system, and scala code should generally avoid using nulls. – Pigmentation

I am no fan of nulls, believe me! In the actual code where I stumbled upon this I was using Trys to integrate with some legacy java code. However, null seems incidental to me here: the aspect I find bizarre (scary?) is that adding more type specificity results in an (unintuitive) runtime exception. – Dobbins

It's a weirdness of the language that putting types in a for/yield results in a pattern match; it's not a problem that would occur with "ordinary" types on expressions. All I can say is it's unfortunate, but probably can't be changed at this stage; I tend to avoid ever giving a type on the left hand side of a for/yield for this reason :/ – Pigmentation

@Pigmentation I think I'll stop typing the LHS too! Thanks! – Dobbins

Answer 1

null is not an instance of anything:

scala> (null: String) match { case _: String => }
scala.MatchError: null
  ... 33 elided

scala> val s: String = null
s: String = null

scala> s.isInstanceOf[String]
res1: Boolean = false

http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html#type-patterns

Type pattern specifies non-null.

One trick for showing the translation is to comment show:

scala> for {
     |   a <- Option("hello")
     |   b: String = f()
     | } yield (a, b) // show
object $read extends scala.AnyRef {
  def <init>() = {
    super.<init>;
    ()
  };
  object $iw extends scala.AnyRef {
    def <init>() = {
      super.<init>;
      ()
    };
    import $line4.$read.$iw.$iw.f;
    object $iw extends scala.AnyRef {
      def <init>() = {
        super.<init>;
        ()
      };
      val res1 = Option("hello").map(((a) => {
        val b: String = f;
        scala.Tuple2(a, b)
      })).map(((x$1) => x$1: @scala.unchecked match {
        case scala.Tuple2((a @ _), (b @ (_: String))) => scala.Tuple2(a, b)
      }))
    }
  }
}
scala.MatchError: (hello,null) (of class scala.Tuple2)
  at $anonfun$2.apply(<console>:10)
  at $anonfun$2.apply(<console>:10)
  at scala.Option.map(Option.scala:145)
  ... 39 elided

Clipboard

That shows what is going wrong, but it's arguably a bug because there's no reason to pattern match on that last map since it by construction already has the correct type (at val b: String = f). (It's also arguably a bug that the identity function isn't optimized out, but we can imagine a more complex case.) – Chromosome

@RexKerr yes, arguably. It's a stronger arg if it used the old val b: String = f() syntax, maybe modulo SI-900 where type pat on generator means filter. But you wouldn't optimize the ordinary match. Maybe we should start writing val i = 42 : Int. Then val s : String = null : String could fail as though wrapped in Tuple1. – Clipboard

Does the language spec dictate that the de-sugaring of the type annotation in the comprehension be implemented as a pattern match? – Dobbins

@bkent314 Yes. Unlike a regular val definition, the LHS is a pattern1. scala-lang.org/files/archive/spec/2.11/… Compare pattern2 in scala-lang.org/files/archive/spec/2.11/… – Clipboard