What's the Scala way to implement a retry-able call like this one?

Question

Still the newbie in Scala and I'm now looking for a way to implement the following code on it:

@Override
public void store(InputStream source, String destination, long size) {

    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentLength(size);
    final PutObjectRequest request = new PutObjectRequest(
            this.configuration.getBucket(), destination, source, metadata);

    new RetryableService(3) {

        @Override
        public void call() throws Exception {
            getClient().putObject(request);
        }
    };

}

What would be the best way to implement the same funcionality that RetryableService implements but in Scala?

It basically calls the call method N times, if all of them fail the exception is then raised, if they succeed it moves on. This one does not return anything but then I have another version that allows for returning a value (so, i have two classes in Java) and I believe I could do with a single class/function in Scala.

Any ideas?

EDIT

Current implementation in java is as follows:

public abstract class RetryableService {

private static final JobsLogger log = JobsLogger
        .getLogger(RetryableService.class);

private int times;

public RetryableService() {
    this(3);
}

public RetryableService(int times) {
    this.times = times;
    this.run();
}

private void run() {

    RuntimeException lastExceptionParent = null;

    int x = 0;

    for (; x < this.times; x++) {

        try {
            this.call();
            lastExceptionParent = null;
            break;
        } catch (Exception e) {
            lastExceptionParent = new RuntimeException(e);
            log.errorWithoutNotice( e, "Try %d caused exception %s", x, e.getMessage() );

            try {
                Thread.sleep( 5000 );
            } catch (InterruptedException e1) {
                log.errorWithoutNotice( e1, "Sleep inside try %d caused exception %s", x, e1.getMessage() );
            }

        }

    }

    try {
        this.ensure();
    } catch (Exception e) {
        log.error(e, "Failed while ensure inside RetryableService");
    }

    if ( lastExceptionParent != null ) {
        throw new IllegalStateException( String.format( "Failed on try %d of %s", x, this ), lastExceptionParent);
    }   

}

public void ensure() throws Exception {
    // blank implementation
}

public abstract void call() throws Exception;

}

Serviceman

Answer 1

Recursion + first class functions by-name parameters == awesome.

def retry[T](n: Int)(fn: => T): T = {
  try {
    fn
  } catch {
    case e =>
      if (n > 1) retry(n - 1)(fn)
      else throw e
  }
}

Usage is like this:

retry(3) {
  // insert code that may fail here
}

Edit: slight variation inspired by @themel's answer. One fewer line of code :-)

def retry[T](n: Int)(fn: => T): T = {
  try {
    fn
  } catch {
    case e if n > 1 =>
      retry(n - 1)(fn)
  }
}

Edit Again: The recursion bothered me in that it added several calls to the stack trace. For some reason, the compiler couldn't optimize tail recursion in the catch handler. Tail recursion not in the catch handler, though, optimizes just fine :-)

@annotation.tailrec
def retry[T](n: Int)(fn: => T): T = {
  val r = try { Some(fn) } catch { case e: Exception if n > 1 => None }
  r match {
    case Some(x) => x
    case None => retry(n - 1)(fn)
  }
}

Edit yet again: Apparently I'm going to make it a hobby to keep coming back and adding alternatives to this answer. Here's a tail-recursive version that's a bit more straightforward than using Option, but using return to short-circuit a function isn't idiomatic Scala.

@annotation.tailrec
def retry[T](n: Int)(fn: => T): T = {
  try {
    return fn
  } catch {
    case e if n > 1 => // ignore
  }
  retry(n - 1)(fn)
}

Scala 2.10 update. As is my hobby, I revisit this answer occasionally. Scala 2.10 as introduced Try, which provides a clean way of implementing retry in a tail-recursive way.

// Returning T, throwing the exception on failure
@annotation.tailrec
def retry[T](n: Int)(fn: => T): T = {
  util.Try { fn } match {
    case util.Success(x) => x
    case _ if n > 1 => retry(n - 1)(fn)
    case util.Failure(e) => throw e
  }
}

// Returning a Try[T] wrapper
@annotation.tailrec
def retry[T](n: Int)(fn: => T): util.Try[T] = {
  util.Try { fn } match {
    case util.Failure(_) if n > 1 => retry(n - 1)(fn)
    case fn => fn
  }
}

Vacancy

Actually, by-name parameters, though I'm concerned there might be something awkward going on when passing a by-name parameter recursively. – Shanly

If you still have the habit @dave take a look at the scala.util.Try object! Great answers! – Glynas

I'll just point that sometimes you will want to retry only on some exceptions while directly throwing others. You can use scala.util.control.Exception.catching(classOf[E1],classOf[E2]).withTry(fn) match... – Kendrickkendricks

@annotation.tailrec def retry[T](n: Int)(fn: => T): Try[T] = { Try { fn } match { case x: Success[T] => x case Failure(NonFatal(_)) if n > 1 => retry(n - 1)(fn) case f => f } } – Synchrocyclotron

Why is scala.util.control.Exception.catching(classOf[E1],classOf[E2]) of type Catch[Nothing] ? E.g. Exception.catching(classOf[PSQLException]) becomes type : Catch[Nothing]. Is ut supposed to be that way? I don't know why they added type-parameterization to it if it's always gonna be Nothing anyway...? :/ – Hell

How about a little lesser lines :) @annotation.tailrec def retry[T](n: Int)(fn: => T): Try[T] = Try(fn) match { case x: util.Success[T] => x case _ if n > 1 => retry(n - 1)(fn) case f => f } – Ingratiating

I read all your solutions and you could really notice how each improves the previous. This is why Scala is love. This is what makes scala great. Beautiful. I am adapting the last one and adding escalating timeouts between each retry. I will try to keep the beauty as much as possible. – Ariana

Can this be modified to retry only for certain exceptions? – Tennes

@IshankGulati Change the retry case in the catch clause to specify the exception you want to retry on. Like case util.Failure(e: RuntimeException) if n > 1 => retry(n - 1)(fn) in the Scala 2.10 example. – Vacancy

@Vacancy but then retry function will become specific to a particular request. I was thinking of passing vararg list of exceptions in method. – Tennes

How can it be improved to add a waiting time between each retry? – Embolism

@Embolism I think just calling Thread.sleep(waitMillis) before retry(n - 1)(fn) would do the trick. – Vacancy

Is there any way to make tailrec works with recoverWith so I only have to retry on a specific exception? – Snowwhite

one question here. How to modify it if my function returns Future? – Kohlrabi

@GauravAbbi what does the case f => f handle? is that when Try(fn) returns Failure[T]? if so, what is the difference with case _ then? – Eversion

Can I ask what's the purpose of returning a Try[T]? I'm new to Scala and I don't see how the usage will work? If I call retry({some code block}), then the return value will be Try[{...return value of code block...}]? What does this mean? – Bitolj

@user1886304 Using Try[T] is an alternative to throwing an exception that tends to work better with a functional programming paradigm. There's more info in the ScalaDocs: scala-lang.org/api/2.9.3/scala/util/Try.html – Vacancy

How can I convert this if fn returns a future? – Fresh

Doesn't the tail recursive function need the final modifier? Also I'm trying to figure out how to add a method like this to an implicit class for code blocks implicit class CodeBlockUtils[R] (codeBlock: => R) { – Moir

Answer 2

There is a method in scalaz.concurrent.Task[T]: http://docs.typelevel.org/api/scalaz/nightly/#scalaz.concurrent.Task

def retry(delays: Seq[Duration], p: (Throwable) ⇒ Boolean = _.isInstanceOf[Exception]): Task[T]

Given a Task[T], you can create a new Task[T] which will retry a certain number of times, where the delay between retries is defined by the delays parameter. e.g.:

// Task.delay will lazily execute the supplied function when run
val myTask: Task[String] =
  Task.delay(???)

// Retry four times if myTask throws java.lang.Exception when run
val retryTask: Task[String] =
  myTask.retry(Seq(20.millis, 50.millis, 100.millis, 5.seconds))

// Run the Task on the current thread to get the result
val result: String = retryTask.run

Catenate

Answer 3

Here is one possible implementation:

def retry[T](times: Int)(fn: => T) = 
    (1 to times).view flatMap (n => try Some(fn) catch {case e: Exception => None}) headOption

You can use it like this:

retry(3) {
    getClient.putObject(request)
}

retry also returns Some[T] if body was processed successfully and None if body was only throwing exceptions.

---

Update

If you want to bobble up last exception, then you can take very similar approach but use Either instead of Option:

def retry[T](times: Int)(fn: => T) = {
    val tries = (1 to times).toStream map (n => try Left(fn) catch {case e: Exception => Right(e)}) 

    tries find (_ isLeft) match {
        case Some(Left(result)) => result
        case _ => throw tries.reverse.head.right.get
    }
}

Also, as you can see, at the end, instead of having only last exception, I have them all. So you can also wrap them in some AggregatingException if you want and then throw it. (for simplicity, I just throw last exception)

Lucretialucretius

If it fails all times the exception must bubble up, I'll add the full implementation in the question. – Parting

Note that this is probably not what the OP intends re: side effects - retry(3) { println("foo") } will print three lines. – Brewer

@themel: With my implementation of retry, retry(3) {println("foo")} will print only once – Lucretialucretius

Did you try? It doesn't for me in Scala 2.8.1. – Brewer

@themel: I use 2.9.1.final and it prints foo only one time – Lucretialucretius

@tenshi: Right, I can reproduce - works in 2.9.1, but not in 2.8.1. I didn't expect that. – Brewer

Answer 4

I'd suggest this -

def retry[T](n: Int)(code: => T) : T = { 
  var res : Option[T] = None
  var left = n 
  while(!res.isDefined) {
    left = left - 1 
    try { 
      res = Some(code) 
    } catch { 
      case t: Throwable if left > 0 => 
    }
  } 
  res.get
} 

It does:

scala> retry(3) { println("foo"); }
foo

scala> retry(4) { throw new RuntimeException("nope"); }
java.lang.RuntimeException: nope
        at $anonfun$1.apply(<console>:7)
        at $anonfun$1.apply(<console>:7)
        at .retry(<console>:11)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at RequestResult$.<init>(<console>:9)
        at RequestResult$.<clinit>(<console>)
        at RequestResult$scala_repl_result(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at scala.tools.nsc.Interpreter$Request$$anonfun$loadAndRun$1$$anonfun$apply$17.apply(Interpreter.scala:988)
        at scala.tools.nsc.Interpreter$Request$$anonfun$loadAndRun$1$$anonfun$apply$17.apply(Interpreter....
scala> var i = 0 ;
i: Int = 0

scala> retry(3) { i = i + 1; if(i < 3) throw new RuntimeException("meh");}

scala> i
res3: Int = 3

It can probably be improved to be more idiomatic Scala, but I am not a big fan of one-liners that require the reader to know the entire standard library by heart anyways.

Brewer

Answer 5

You can express the idea in functional style using scala.util.control.Exception:

@annotation.tailrec
def retry[T](n: Int)(fn: => T): T =
  Exception.allCatch.either(fn) match {
    case Right(v)             => v;
    case Left(e) if (n <= 1)  => throw e;
    case _                    => retry(n - 1)(fn);
  }

As we can see, tail recursion can be used here.

This approach gives you the additional benefit that you can parametrize the catch container, so you can only retry a certain subset of exceptions, add finalizers etc. So the final version of retry might look like:

/** Retry on any exception, no finalizers. */
def retry[T](n: Int)(fn: => T): T =
  retry(Exception.allCatch[T], n)(fn);

/** Parametrized retry. */
@annotation.tailrec
def retry[T](theCatch: Exception.Catch[T], n: Int)(fn: => T): T =
  theCatch.either(fn) match {
    case Right(v)             => v;
    case Left(e) if (n <= 1)  => throw e;
    case _                    => retry(theCatch, n - 1)(fn);
  }

With this, you can do complex stuff like:

retry(Exception.allCatch andFinally { print("Finished.") }, 3) {
  // your scode
}

Sherrylsherurd

Answer 6

There is an existing library that can help with that, called retry, and there is a Java library too, called guava-retrying.

Here are some examples of using retry:

// retry 4 times
val future = retry.Directly(4) { () => doSomething }

// retry 3 times pausing 30 seconds in between attempts
val future = retry.Pause(3, 30.seconds) { () => doSomething }

// retry 4 times with a delay of 1 second which will be multipled
// by 2 on every attempt
val future = retry.Backoff(4, 1.second) { () => doSomething }

Unaccountable

Answer 7

I like the accepted solution, but suggest checking the exception is NonFatal:

// Returning T, throwing the exception on failure
@annotation.tailrec
def retry[T](n: Int)(fn: => T): T = {
  Try { fn } match {
    case Success(x) => x
    case _ if n > 1 && NonFatal(e) => retry(n - 1)(fn)
    case Failure(e) => throw e
  }
}

You don't want to retry a control flow exception, and usually not for thread interrupts...

Careless

The NonFatal match line won't compile unless I do: case Failure(e) if n > 1 && NonFatal(e) => retry(n - 1)(fn). – Jadejaded

Answer 8

If you want control of which exceptions you retry, you can use methods in scala.util.control.Exception:

import java.io._
import scala.util.control.Exception._

def ioretry[T](n: Int)(t: => T) = (
  Iterator.fill(n){ failing[T](classOf[IOException]){ Option(t) } } ++
  Iterator(Some(t))
).dropWhile(_.isEmpty).next.get

(As written, it will also retry on null; that's the Option(t) part. If you want nulls to be returned, use Some(t) inside the iterator fill instead.)

Let's try this out with

class IoEx(var n: Int) {
  def get = if (n>0) { n -= 1; throw new IOException } else 5
}
val ix = new IoEx(3)

Does it work?

scala> ioretry(4) { ix.get }
res0: Int = 5

scala> ix.n = 3

scala> ioretry(2) { ix.get }
java.io.IOException
    at IoEx.get(<console>:20)
    ...

scala> ioretry(4) { throw new Exception }
java.lang.Exception
    at $anonfun$1.apply(<console>:21)
    ...

Looks good!

Inset

Answer 9

I ended up adapting a previous answer to allow filtering on which exceptions to retry on:

  /**
   * Attempt 'fn' up to 'attempts' times, retrying only if 'forExceptions' returns true for retry-able exceptions.
   */
  def retry[T](attempts: Int, forExceptions: (Throwable) => Boolean)(fn: => T): T =
  {
    // toStream creates a lazily evaluated list, which we map to a try/catch block resulting in an Either
    val tries = (1 to attempts).toStream map
      {
        n =>
          try
            Left(fn)
          catch
            {
              case e if forExceptions(e) => Right(e)
            }
      }

    // find the first 'Either' where left is defined and return that, or if not found, return last
    // exception thrown (stored as 'right').  The cool thing is that because of lazy evaluation, 'fn' is only
    // evaluated until it success (e.g., until Left is found)
    tries find (_ isLeft) match
    {
      case Some(Left(result)) => result
      case _ => throw tries.reverse.head.right.get
    }

  }

You can call in two ways:

val result = retry(4, _.isInstanceOf[SomeBadException])
{
   boom.doit()
}

or with partial functions (also showing version where don't care about return value)

    def pf: PartialFunction[Throwable, Boolean] =
    {
      case x: SomeOtherException => true
      case _ => false
    }

   retry(4, pf)
   {
      boom.doit()
   }

Montagnard

Answer 10

This solution is not optimized by compiler to tail recursion for some reason (who knows why?), but in case of rare retries would be an option:

def retry[T](n: Int)(f: => T): T = {
  Try { f } recover {
    case _ if n > 1 => retry(n - 1)(f)
  } get
}

Usage:

val words: String = retry(3) {
  whatDoesTheFoxSay()
}

End of the answer. Stop reading here

---

Version with result as a Try:

def reTry[T](n: Int)(f: => T): Try[T] = {
  Try { f } recoverWith {
    case _ if n > 1 => reTry(n - 1)(f)
  }
}

Usage:

// previous usage section will be identical to:
val words: String = reTry(3) {
  whatDoesTheFoxSay()
} get

// Try as a result:
val words: Try[String] = reTry(3) {
  whatDoesTheFoxSay()
}

Version with a function returning Try

def retry[T](n: Int)(f: => Try[T]): Try[T] = {
  f recoverWith {
    case _ if n > 1 => reTry(n - 1)(f)
  }
}

Usage:

// the first usage section will be identical to:
val words: String = retry(3) {
  Try(whatDoesTheFoxSay())
} get

// if your function returns Try:
def tryAskingFox(): Try = Failure(new IllegalStateException)

val words: Try[String] = retry(3) {
    tryAskingFox()
}

Ankerite

Answer 11

A reusable object/method with a pause between attempts:

Retry(3, 2 seconds) { /* some code */ }

Code:

object Retry {
  def apply[A](times: Int, pause: Duration)(code: ⇒ A): A = {
    var result: Option[A] = None
    var remaining = times
    while (remaining > 0) {
      remaining -= 1
      try {
        result = Some(code)
        remaining = 0
      } catch {
        case _ if remaining > 0 ⇒ Thread.sleep(pause.toMillis)
      }
    }
    result.get
  }
}

Benumb

Answer 12

This project seems to provide some nice implementations for different retry mechanisms https://github.com/hipjim/scala-retry

// define the retry strategy

implicit val retryStrategy =
    RetryStrategy.fixedBackOff(retryDuration = 1.seconds, maxAttempts = 2)

// pattern match the result

val r = Retry(1 / 1) match {
    case Success(x) => x
    case Failure(t) => log("I got 99 problems but you won't be one", t)
}

Stern

Answer 13

//Here is one using Play framework

def retry[T](times:Int)(block: => Future[T])(implicit ctx: ExecutionContext):Future[T] = {

type V = Either[Throwable,T]
val i:Iterator[Future[Option[V]]] = 
  Iterator.continually(block.map(t => Right(t)).recover { case e => Left(e) }.map(t => Some(t)))
def _retry:Iteratee[V,V] = {
    def step(ctr:Int)(i:Input[V]):Iteratee[V,V] = i match {
        case Input.El(e) if (e.isRight) => Done(e,Input.EOF)
        case _ if (ctr < times) => Cont[V,V](i => step(ctr + 1)(i))
        case Input.El(e) => Done(e,Input.EOF)
    }
    Cont[V,V](i => step(0)(i))
}
Enumerator.generateM(i.next).run(_retry).flatMap { _ match {
  case Right(t) => future(t)
  case Left(e) => Future.failed(e)
}}
}

Bullnose

Answer 14

Minor improvement to printout attempt x of N

// Returning T, throwing the exception on failure
      @annotation.tailrec
      final def retry[T](n: Int, name: String ="", attemptCount:Int = 1)(fn: => T): T = {
        logger.info(s"retry count: attempt $attemptCount of $n ....... function: $name")
        try {
          val result = fn
          logger.info(s"Succeeded: attempt $attemptCount of $n ....... function: $name")
          result
        } catch {
          case e: Throwable =>
            if (n < attemptCount) { Thread.sleep(5000 * attemptCount); retry(n, name, attemptCount+1)(fn) }
            else throw e 
        }
      }

Condonation