Scala's "for comprehension" with futures

Question

I am reading through the Scala Cookbook (http://shop.oreilly.com/product/0636920026914.do)

There is an example related to Future use that involves for comprehension.

So far my understanding about for comprehension is when use with a collection it will produce another collection with the same type. For example, if each futureX is of type Future[Int], the following should also be of type Future[Int]:

for {
   r1 <- future1
   r2 <- future2
   r3 <- future3
} yield (r1+r2+r3)

Could someone explain me what exactly happening when use <- in this code? I know if it was a generator it will fetch each element by looping.

Oligosaccharide

Answer 1

First about for comprehension. It was answered on SO many many times, that it's an abstraction over a couple of monadic operations: map, flatMap, withFilter. When you use <-, scalac desugars this lines into monadic flatMap:

r <- monad into monad.flatMap(r => ... )

it looks like an imperative computation (what a monad is all about), you bind a computation result to the r. And yield part is desugared into map call. Result type depends on the type of monad's.

Future trait has a flatMap and map functions, so we can use for comprehension with it. In your example can be desugared into the following code:

future1.flatMap(r1 => future2.flatMap(r2 => future3.map(r3 => r1 + r2 + r3) ) )

Parallelism aside

It goes without saying that if execution of future2 depends on r1 then you can't escape sequential execution, but if the future computations are independent, you have two choices. You can enforce sequential execution, or allow for parallel execution. You can't enforce the latter, as the execution context will handle this.

val res = for {
   r1 <- computationReturningFuture1(...)
   r2 <- computationReturningFuture2(...)
   r3 <- computationReturningFuture3(...)
} yield (r1+r2+r3)

will always run sequentially. It can be easily explained by the desugaring, after which the subsequent computationReturningFutureX calls are only invoked inside of the flatMaps, i.e.

computationReturningFuture1(...).flatMap(r1 => 
    computationReturningFuture2(...).flatMap(r2 => 
        computationReturningFuture3(...).map(r3 => r1 + r2 + r3) ) )

However this is able to run in parallel and the for comprehension aggregates the results:

val future1 = computationReturningFuture1(...)
val future2 = computationReturningFuture2(...)
val future3 = computationReturningFuture3(...)

val res = for {
   r1 <- future1
   r2 <- future2
   r3 <- future3
} yield (r1+r2+r3)

Taynatayra

The Scala Language Spec (scala-lang.org/files/archive/nightly/pdfs/ScalaReference.pdf, now a little out of date) gives the precise rules for desugaring of for comprehensions on pages 89-90. – Ligetti

@Alexlv Sorry ,what is monodic operations ?? seems like its something to do with collections by looking at your explanation. – Oligosaccharide

@Oligosaccharide A concept of a monad is more general, so no, it has nothing to do with collections in particular. A monadic function is a function of type A => M[A], so it's a function which gives us a monadic value, which is a value in some context which describes computation, i.e Future which describes a context in which computations are deferred for future. – Taynatayra

@AlexIv sorry to ask this stupid question , what exactly "monadic" means ? I got to know form you two variations i.e monodic functions and monodic value . But still "monodic" greek term is there. Could you please tell me what is "monodic" in plain English or direct me to a resource to understand the concept in programming context ? before I asked this from you I tried gooling but finally got more confused :( – Oligosaccharide

@Oligosaccharide That's a really too big idea and it was discussed too many times. Monadic is just about a concept of a Monad and computation flow. To understand monads i would highly recomend this book and this tutorial or just look for similar questions on StackOverflow – Taynatayra

@AlexIv by the way I purchased the early edition of manning.com/bjarnason, but it does not explain much what is Monad in detail than bunch of examples that hard to understand for a beginner. And in the tutorial it refereed to swiss.csail.mit.edu/classes/6.001/abelson-sussman-lectures this. In that book I searched for monad keyword, but non there :( – Oligosaccharide

@Oligosaccharide if you have bought a MEAP version, the manning sends you a fresh version after each update, check your mailbox or download the latest version. The latest is 11. It contains all chapters, 11 and 12 chapters is just what you need – Taynatayra

The comment, "BUT you should remember that this gonna be evaluated sequentially, not in parallel, because the result of result2 and result3 depends on r1" is somewhat misleading. Whether or not the underlying operation of future2 depends on the outcome of future1 it will be begun in the onComplete of future1 and thus be sequential. Future(1).map(_ => Future(2)) is sequential although the underlying functions are independent. – Intestate

Is the Future thus created is complete ? I tried something similar , it created a future ,tat was not complete – Appointed

This seems a very complex answer to what is really a very simple question. Yes, this describes the implementation well, but most of the time someone looking for the answer won't care about that. – Born

Could you add the desugared version of the sequential example for clarity? – Barrick

@DavidMoles please check the answer, there's already an example of desugared code – Taynatayra

Hi @4lex1v, I see the desugared code for one case, but I don't understand how the other case desugars differently. – Barrick

I've made an edit that explicitly shows the desugared code for the sequential case, hope that helps. – Buggs

Too bad though, because the code looks more compact when the future is created inside for. – Graeae

How to make it parallel – Paravane

Answer 2

To elaborate those existing answers here a simple result to demonstrate how for comprehension works.

Its bit lengthy functions yet they worth taking look into it.

A function that give us a range of integers

scala> def createIntegers = Future{
             println("INT "+ Thread.currentThread().getName+" Begin.")
             val returnValue = List.range(1, 256)
             println("INT "+ Thread.currentThread().getName+" End.")
             returnValue
         }
createIntegers: createIntegers: scala.concurrent.Future[List[Int]]

A function that give us a range of chars

scala> def createAsciiChars = Future{
             println("CHAR "+ Thread.currentThread().getName+" Begin.")
             val returnValue = new ListBuffer[Char]
             for (i <- 1 to 256){
                  returnValue += i.toChar
             }
             println("CHAR "+ Thread.currentThread().getName+" End.")
             returnValue
          }
createAsciiChars: scala.concurrent.Future[scala.collection.mutable.ListBuffer[Char]]

Using these function calls within the for comprehension.

scala> val result = for{
                        i <- createIntegers
                        s <- createAsciiChars
                    } yield i.zip(s)
       Await.result(result, Duration.Inf)
result: scala.concurrent.Future[List[(Int, Char)]] = Future(<not completed>)

For these below lines we can make out that all the function calls are synchronous i.e. createAsciiChars function call is not executed until createIntegers completes its execution.

scala> INT scala-execution-context-global-27 Begin.
       INT scala-execution-context-global-27 End.
       CHAR scala-execution-context-global-28 Begin.
       CHAR scala-execution-context-global-28 End.

Making these function createAsciiChars, createIntegers calls outside the for comprehensions will be asynchronous execution.

Mcdade

Answer 3

It allows r1, r2, r3 to run in parallel, if possible. It may not be possible, depending things like how many threads are available to execute Future computations, but by using this syntax you are telling the compiler to run these computations in parallel if possible, then execute the yield() when all have completed.

Born

The for comprehension doesn't allow them to run in parallel. They are running in parallel because the Futures were defined outside of the for comprehension, but if you were creating Futures inside of the for comprehension, they would run sequentially (ex: r1 <- Future(value1)). Sometimes you want them to run sequentially, for instance, you may have a future that gives you back a an id, which you need to make another call with, which is also a Future. – Alimentary