Concatenate many Future[Seq] into one Future[Seq]

Question

Without Future, that's how I combine all smaller Seq into one big Seq with a flatmap

category.getCategoryUrlKey(id: Int):Seq[Meta] // main method
val appDomains: Seq[Int]

val categories:Seq[Meta] = appDomains.flatMap(category.getCategoryUrlKey(_))

Now the method getCategoryUrlKey could fail. I put a circuit breaker in front to avoid to call it for the next elements after an amount of maxFailures. Now the circuit breaker doesn't return a Seq but a Future[Seq]

lazy val breaker = new akka.pattern.CircuitBreaker(...)

private def getMeta(appDomainId: Int): Future[Seq[Meta]] = {
  breaker.withCircuitBreaker {
    category.getCategoryUrlKey(appDomainId)
  }
}

How to iterate through the List appDomains and combine the result into one single Future[Seq] , possible into Seq ?

If Functional Programming is applicable, is there a way to directly transform without temporary variables ?

Antisepsis

Answer 1

Squash seq of futures using Future.sequence

Future.sequenceconverts Seq[Future[T]] to Future[Seq[T]]

In your case T is Seq. After the sequence operation, you will end up with Seq[Seq[T]]. So Just flatten it after the sequence operation using flatten.

def squashFutures[T](list: Seq[Future[Seq[T]]]): Future[Seq[T]] =
  Future.sequence(list).map(_.flatten)

Your code becomes

Future.sequence(appDomains.map(getMeta)).map(_.flatten)

Mold

Ok , I have to store first into Seq[Future[Seq[T]]] , then I can squashFutures . Is there a way to directly transform without temporary variables ? – Antisepsis

@RaymondChenon Future.sequence(appDomains.map(getMeta)).map(_.flatten) is what you have to do. There are no temp variables here – Mold

You sir are a hero. – Pankey

Answer 2

From TraversableOnce[Future[A]] to Future[TraversableOnce[A]]

val categories = Future.successful(appDomains).flatMap(seq => {
    val fs = seq.map(i => getMeta(i))
    val sequenced = Future.sequence(fs)
    sequenced.map(_.flatten)
})

• Future.successful(appDomains) lifts the appDomains into the context of Future

Hope this helps.

Rudolf

Answer 3

val metaSeqFutureSeq = appDomains.map(i => getMeta(i))
// Seq[Future[Seq[Meta]]]

val metaSeqSeqFuture = Future.sequence(metaSeqFutureSeq)
// Future[Seq[Seq[Meta]]]
// NOTE :: this future will fail if any of the futures in the sequence fails

val metaSeqFuture = metaSeqSeqFuture.map(seq => seq.flatten)
// Future[Seq[Meta]]

If you want to reject the only failed futures but keep the successful one's then we will have to be a bit creative and build our future using a promise.

import java.util.concurrent.locks.ReentrantLock

import scala.collection.mutable.ArrayBuffer
import scala.concurrent.{Future, Promise}
import scala.util.{Failure, Success}

def futureSeqToOptionSeqFuture[T](futureSeq: Seq[Future[T]]): Future[Seq[Option[T]]] = {
  val promise = Promise[Seq[Option[T]]]()

  var remaining = futureSeq.length

  val result = ArrayBuffer[Option[T]]()
  result ++ futureSeq.map(_ => None)

  val resultLock = new ReentrantLock()

  def handleFutureResult(option: Option[T], index: Int): Unit = {
    resultLock.lock()
    result(index) = option
    remaining = remaining - 1
    if (remaining == 0) {
      promise.success(result)
    }
    resultLock.unlock()
  }

  futureSeq.zipWithIndex.foreach({ case (future, index) => future.onComplete({
    case Success(t) => handleFutureResult(Some(t), index)
    case Failure(ex) => handleFutureResult(None, index)
  }) })

  promise.future
}

val metaSeqFutureSeq = appDomains.map(i => getMeta(i))
// Seq[Future[Seq[Meta]]]

val metaSeqOptionSeqFuture = futureSeqToOptionSeqFuture(metaSeqFutureSeq)
// Future[Seq[Option[Seq[Meta]]]]

val metaSeqFuture = metaSeqSeqFuture.map(seq => seq.flatten.flatten)
// Future[Seq[Meta]]

Yelena