Best Scala imitation of Groovy's safe-dereference operator (?.)?

Question

I would like to know what the best Scala imitation of Groovy's safe-dereference operator (?.), or at least some close alternatives are?

I've discussed it breifly on Daniel Spiewak's blog, but would like to open it up to StackOverFlow...

For the sake of everyone's time, here is Daniel's initial response, my counter, and his 2nd response:

@Antony

Actually, I looked at doing that one first. Or rather, I was trying to replicate Ragenwald’s andand “operator” from Ruby land. The problem is, this is a bit difficult to do without proxies. Consider the following expression (using Ruby’s andand, but it’s the same with Groovy’s operator):

test.andand().doSomething()

I could create an implicit conversion from Any => some type implementing the andand() method, but that’s where the magic stops. Regardless of whether the value is null or not, the doSomething() method will still execute. Since it has to execute on some target in a type-safe manner, that would require the implementation of a bytecode proxy, which would be flaky and weird (problems with annotations, final methods, constructors, etc).

A better alternative is to go back to the source of inspiration for both andand as well as Groovy’s safe dereference operator: the monadic map operation. The following is some Scala syntax which uses Option to implement the pattern:

val something: Option[String] = … // presumably could be either Some(…) or None

val length = something.map(_.length)

After this, length either be Some(str.length) (where str is the String object contained within the Option), or None. This is exactly how the safe-dereferencing operator works, except it uses null rather than a type-safe monad.

As pointed out above, we could define an implicit conversion from some type T => Option[T] and then map in that fashion, but some types already have map defined, so it wouldn’t be very useful. Alternatively, I could implement something similar to map but with a separate name, but any way it is implemented, it will rely upon a higher-order function rather than a simple chained call. It seems to be just the nature of statically typed languages (if anyone has a way around this, feel free to correct me).

Daniel Spiewak Monday, July 7, 2008 at 1:42 pm

My 2nd question:

Thanks for the response Daniel regarding ?. I think I missed it! I think I understand what you’re proposing, but what about something like this, assuming you don’t have control over the sources:

company?.getContactPerson?.getContactDetails?.getAddress?.getCity

Say it’s a java bean and you can’t go in and change the return values to Something[T] - what can we do there?

Antony Stubbs Tuesday, July 21, 2009 at 8:07 pm oh gosh - ok on re-read that’s where you’re proposing the implicit conversion from T to Option[T] right? But would you still be able to chain it together like that? You’d still need the map right? hmm….

var city = company.map(_.getContactPerson.map(_.getContactDetails.map(_.getAddress.map(_.getCity))))

?

Antony Stubbs Tuesday, July 21, 2009 at 8:10 pm

His 2nd response:

@Antony

We can’t really do much of anything in the case of company?.getContactPerson, etc… Even assuming this were valid Scala syntax, we would still need some way to prevent the later calls in the chain. This is not possible if we’re not using function values. Thus, something like map is really the only option.

An implicit conversion to Option wouldn’t be bad, but by making things implicit, we’re circumventing some of the protection of the type system. The best way to do this sort of thing is to use for-comprehensions in concert with Option. We can do map and flatMap, but it’s much nicer with magical syntax:

 for {
   c < - company
   person <- c.getContactPerson   
   details <- person.getContactDetails
   address <- details.getAddress 
  } yield address.getCity

Daniel Spiewak Tuesday, July 21, 2009 at 9:28 pm

P.s. if Daniel posts his original answers on his blog as answers, I will edit the question to remove them for the sake of the System.

Battleax

See also #1364861 – Consul

Example of the most beautiful and powerful language (Scala) with most ugly construct (all these for comprehension, chained maps, etc., all of these) in the world. What a pity! Hope Scala 3 with meta-programming magic will change this! – Madalene

I found this would be useful: contributors.scala-lang.org/t/… . It mentioned a Dsl.scala lib which is written by Thoughtworks can unifies monads, generators, asynchronous functions, coroutines and continuations to a single universal syntax looks interesting. – Madalene

Answer 1

How about this?

def ?[A](block: => A) =
  try { block } catch {
    case e: NullPointerException if e.getStackTrace()(2).getMethodName == "$qmark" => null
    case e => throw e
  }

Using this little snippet, you can dereference safely and the code itself is quite succinct:

val a = ?(b.c.d.e)

a == null if b or b.c or b.c.d or b.c.d.e is null, otherwise, a == b.c.d.e

I think the value of a safe-dereference operator is diminished when you are using a language like Scala which has facilities like call-by-name and implicits.

ps: I modify the code above a bit in light of one of the comments below to handle the case when NullPointerException is actually thrown inside the called function.

BTW, I think using the function below is a more idiomatic way of writing Scala:

def ??[A](block: => A): Option[A] = ?(block) match {
    case a: A => Some(a)
    case _ => None
  }

like so:

??(a.b.c.d) match {
    case Some(result) => // do more things with result
    case None => // handle "null" case
  }

Gerius

Of course! Beautiful! Yet again I am seriously impressed with Scala! This is even better than the Groovy operator! I wonder if you can do the same with closures in Groovy? Definitely gonna add this one to the tool box! To appear in Wicket-Scala extensions soon... Thanks! – Battleax

A demo: scala> def ?[A](block: => A) = | try { block } catch { case e: NullPointerException => null } $qmark: [A](=> A)Any scala> case class Address(city:String) scala> case class Person(a:Address) scala> case class Company(p:Person) scala> var a = Company(Person(Address("Customs st"))) a: Company = Company(Person(Address(Customs st))) scala> var b = Company(Person(null)) b: Company = Company(Person(null)) scala> println(a.p.a.city) Customs st scala> println(b.p.a.city) java.lang.NullPointerException scala> println(?(b.p.a.city)) null – Battleax

perhaps you could edit your comment and add that demo in, but with line breaks? – Battleax

This is kinda crap if one of the method calls actually throws an NPE. – Locus

What if you catch a nested exception from inside another method called $qmark? :P – Locus

For comprehension is a safer option, really. – Locus

the 3rd stack frame is THE stack frame for this function. as long as the function is called "?", we are pretty safe. – Gerius

The stack frame thing is ingenius. Using reflection can make the code much slower, though. This is rather safe, but using it on a critical path can be unexpectedly unpleasant. – Amaurosis

However - it only uses reflection when an actual exception is thrown - so in that case we don't usually care about performance :) – Battleax

This is awful. Beyond awful. You're trapping what might be legitimate NPEs firstly, and secondly, ignoring the fact that an operation might have a side effect before it throws an NPE. – Casarez

This solution is not exhaustive enough. I had a case when this code did not work. The thing was on android platform and looked something like: ??(getIntent.getExtras.getBoolean(...)) match { case Some(result) => doSomething case None => doSomethingElse } The qmark method was there in the stacktrace. In the forth place though. Not the third. So please update your code. The stactrace thing is clever but scala can be tricky when it comes to method invocation. – Degradable

Seriously guys, did anyone think of how expensive exceptions are ? It is easily a couple of orders of magnitude slower. so, to suggest this is egregious. . – Ileenileitis

With Scala 2.10, it is possible to implement this idea using macros. I played a little with a macro that simply adds necessary nullguards and it seems to work. Such solution doesn't need to deal with NPEs so it doesn't have any of the problems mentioned. I'll try to refine the code a little and maybe post some it here. – Proverbs

Answer 2

There are two things that need to be considered here.

First, there is the problem of the "nothing". How do you chain things when a part of the chain may not return anything? The answer is using Option and for comprehensions. For example:

scala> case class Address(city: Option[String] = None, street: Option[String] = None, number: Option[Int] = None)
defined class Address

scala> case class Contact(name: String, phone: Option[String] = None, address: Option[Address] = None)
defined class Contact

scala> case class ContactDetails(phone: Option[String] = None, address: Option[Address] = None)
defined class ContactDetails

scala> case class Contact(phone: Option[String] = None, address: Option[Address] = None)
defined class Contact

scala> case class Person(name: String, contactDetails: Option[Contact] = None)
defined class Person

scala> case class Company(name: String, contactPerson: Option[Person] = None)
defined class Company

scala> val p1 = Company("ABC", Some(Person("Dean", Some(Contact(None, Some(Address(city = Some("New England"))))))))
p1: Company = Company(ABC,Some(Person(Dean,Some(Contact(None,Some(Address(Some(New England),None,None)))))))

scala> val p2 = Company("Finnicky", Some(Person("Gimli", None)))
p2: Company = Company(Finnicky,Some(Person(Gimli,None)))

scala> for(company <- List(p1, p2);
     | contactPerson <- company.contactPerson;
     | contactDetails <- contactPerson.contactDetails;
     | address <- contactDetails.address;
     | city <- address.city) yield city
res28: List[String] = List(New England)

This is how you are supposed to write code which may return something or not in Scala.

The second problem, of course, is that sometimes you may not have access to the source code to do the proper convertion. In this case, there is some additional syntax overhead to be head, unless an implicit can be used. I'll give an example below, in which I use an "toOption" function -- there is such a thing on Scala 2.8, of which I'll talk about below.

scala> def toOption[T](t: T): Option[T] = if (t == null) None else Some(t)
toOption: [T](t: T)Option[T]

scala> case class Address(city: String = null, street: String = null, number: Int = 0)
defined class Address

scala> case class Contact(phone: String = null, address: Address = null)
defined class Contact

scala> case class Person(name: String, contactDetails: Contact = null)
defined class Person

scala> case class Company(name: String, contactPerson: Person = null)
defined class Company

scala> val p1 = Company("ABC", Person("Dean", Contact(null, Address(city = "New England"))))
p1: Company = Company(ABC,Person(Dean,Contact(null,Address(New England,null,0))))

scala> val p2 = Company("Finnicky", Person("Gimli"))
p2: Company = Company(Finnicky,Person(Gimli,null))

scala> for(company <- List(p1, p2);
     | contactPerson <- toOption(company.contactPerson);
     | contactDetails <- toOption(contactPerson.contactDetails);
     | address <- toOption(contactDetails.address);
     | city <- toOption(address.city)) yield city
res30: List[String] = List(New England)

Remember that you can be quite creative in naming a function. So, instead of "toOption", I might have named it "?", in which case I'd write things like "?(address.city)".

Thanks to nuttycom for reminding me, on Scala 2.8 there is an Option factory on the object Option, so I can just write Option(something). In effect, you can replace "toOption" above with "Option". And if you prefer using ?, you can just use import with rename.

Amaurosis

The method you're thinking of in 2.8 is Option.apply(). Thus, just as you have Some(a) and None, you now have Option(a) which returns if (a == null) None else Some(a) – Chon

Given 2.8, you could actually get your "?" by simply doing import scala.{Option => ?} – Chon

This started to be documented in the Options doc since version 2.8.1: scala-lang.org/api/2.8.1/scala/Option.html – Snoopy

Answer 3

Create this implicit conversion.

class SafeDereference[A](obj: A) {
  def ?[B >: Null](function: A => B): B = if (obj == null) null else function(obj)
}

implicit def safeDereference[A](obj: A) = new SafeDereference(obj)

The usage isn't as pretty as Groovy, but it's not awful.

case class Address(state: String)
case class Person(first: String, last: String, address: Address)
val me = Person("Craig", "Motlin", null)

scala> me ? (_.first)
res1: String = Craig

scala> me ? (_.address)
res2: Address = null

scala> me ? (_.address) ? (_.state)
res3: String = null

Heck

Answer 4

Monadic bind (flatMap/map) with the scala.Option type. Support is also provided by for-comprehensions. Scalaz provides an applicative functor style if you prefer.

This is not equivalent, but a far better solution than Groovy's operator for many reasons.

Marxism

An example: Some(5).map(_+5) == Some(10) val foo:Option[Int] = None foo.map(_+5) == None – Corny

...and how is it a far better solution a ?.? – Theresa

Answer 5

Not mine but a coworker's

class NullCoalescer[T <: AnyRef](target: T) {
    def ?? (other: T) =
        if(target == null) other else target
}
object NullCoalescerConversions {
    implicit def toNullCoalescer[T <: AnyRef](target: T): NullCoalescer[T] = 
        new NullCoalescer(target)
}

println (System.getProperty("maybe") ?? "definitely")

Dunleavy

Answer 6

To follow up Daniel C. Sobral's answer, the reason Option is preferred is because idiomatic Scala does not use null pointers. If you can, rewrite the code to return Options instead of nullable references. Chained flatMaps are cleaner than for-comprehensions, since you don't need a new variable name for each step. If all the values are optional (as in the Groovy example), the Scala approach would look like this:

(company flatMap _.getContactPerson
         flatMap _.getContactDetails
         flatMap _.getAddress
         flatMap _.getCity) match {
  case Some(city) => ...
  case None       => ...
}

If you must use nullable values for Java interoperability, here's an approach that gives you safety without NPE-wrangling or too much clutter:

sealed trait Nullable[+A] {
  def apply[B](f:A=>B): Nullable[B]
}

def ?[A](a: A) = a match {
  case null => NullRef
  case _    => Ref(a)
}

case class Ref[A](value: A) extends Nullable[A] {
  def apply[B](f:A=>B) = ?(f(value))
}

object NullRef extends Nullable[Nothing] {
  def apply[B](f: Nothing=>B): Nullable[B] = NullRef
}


?(company)(_.getContactPerson)(_.getContactDetails)(_.getAddress)(_.getCity) match {
  case Ref(city) => ...
  case _         => ...
}

This should be easy to expand to a full Option-style monad if desired.

Lowly

Answer 7

Because this would look terrible as a comment, here's a commented version of Walter's code:

/**
 * Safe dereference operator. E.g. ?(a.b.c.null.dd)
 */
def ?[A](block: => A) = {
  try { block } catch {
    // checks to see if the 3rd to last method called in the stack, is the ?() function, 
    // which means the null pointer exception was actually due to a null object, 
    // otherwise the ?() function would be further down the stack.
    case e: NullPointerException if e.getStackTrace()(2).getMethodName == "$qmark" => {null}
    // for any other NullPointerException, or otherwise, re-throw the exception.
    case e => throw e
  }

And the specification, which passes:

case class Company(employee:Employee)
case class Employee(address:Address){
  def lookupAddressFromDb:Address = throw new NullPointerException("db error")
}
case class Address(city:String)

"NullSafe operater" should {
  "return the leaf value when working with non-null tree" in {
    val company = Company(Employee(Address("Auckland")))
    val result = ?( company.employee.address.city )
    result mustEq "Auckland"
  }
  "return null when working with a null element at some point in the tree" in {
    val company = Company(null)
    val result = ?( company.employee.address.city )
    result must beNull
  }
  "re-throw the NPE when working with a method which actually throws a NullPointerException" in {
    val company = Company(Employee(Address("Auckland")))
    ?( company.employee.lookupAddressFromDb.city ) aka "the null-safe lookup method" must throwA[NullPointerException]
  }   
}

Battleax

Answer 8

I liked Daniel C. Sobral's use of for comprehensions--- it gets to the point more quickly than the cascade of nested matches I had been doing. However, it's still not very convenient because there are still intermediate dummy variables (and too much typing).

We want something like a?.b?.c?.d so we don't have to think about what comes in between: just try to get something and give me an Option in case you can't get it.

For context, suppose I have

case class Inner(z: Option[Int])
case class Outer(y: Option[Inner])
val x = Some(Outer(Some(Inner(Some(123)))))

that I want to unpack. The for comprehension would go like the following

for (tmp1 <- x; tmp2 <- tmp1.y; tmp3 <- tmp2.z) yield tmp3

which results in Some(123). The problem is too many temporary variables (and the fact that it's partially reading backward).

I find it easier to do it with flatMap, like this

x.flatMap(_.y.flatMap(_.z))

or

x flatMap {_.y flatMap {_.z}}

which also results in Some(123).

One could cut down on the verbosity and use the desired ? symbol by effectively giving the Option type a method ? that does the same thing as flatMap. Option is sealed from subclassing, but we can simulate the new method with implicit conversions.

case class OptionWrapper[A](opt: Option[A]) {
  def ?[B](f: (A) => Option[B]): Option[B] = opt.flatMap(f)
}
implicit def toOptionWrapper[T](opt: Option[T]) = OptionWrapper(opt)
implicit def fromOptionWrapper[T](wrap: OptionWrapper[T]) = wrap.opt

And then

x ? {_.y ? {_.z}}

yields Some(123. It's still not perfect because there are nested brackets and underscores that you have to get right, but it's better than any alternatives I've seen.

Rothberg

I just noticed that D. Gates is able to do this without the curly brackets, but my Scala console says error: missing parameter type for expanded function ((x$1) => z.flatMap(x$1.y)) when I try to x flatMap _.y. I often encounter these errors and am forced to add more decorative syntax. – Rothberg

Answer 9

I created a macro-based library to solve exactly this problem:

https://github.com/ryanstull/ScalaNullSafe

The way it works is it decomposes an expression that you pass into it and rewrites it adding in null-checks at each point that there could be a null value.

For example:

import com.ryanstull.nullsafe._

case class A(b: B)
case class B(c: C)
case class C(d: D)
case class D(e: E)
case class E(s: String)

val a = A(B(C(null)))
?(a.b.c.d.e.s) //No NPE! Just returns null

val a2 = A(B(C(D(E("Hello")))))
?(a2.b.c.d.e.s) //Returns "Hello"

What is happening behind the scenes is that ?(a.b.c) gets transformed into

if(a != null){
  val b = a.b
  if(b != null){
    b.c
  } else null
} else null

There are other variants as well such as:

opt: A -> Option[A]
notNull: Boolean

And ?? which acts as a null-coalescing operator.

The project README has more details.

Dore