Akka Configuration Exception: Logger specified can't be loaded

Question

I am trying to load test my spray http client based on an actor model using the apache bench tool. The performance is not great. All the more, when I increase the concurrency level to around 200-300, I see several errors related to loggers.

Code Snippet

   val result = for {
      response <- GlobalObject.io.ask(HttpRequest(GET,URL)).mapTo[spray.http.HttpResponse]
    } yield response
   result Oncomplete{
    case Success(response) => doSomething
   }

Full Stack Trace:

[info] [WARN] [03/31/2015 14:31:52.580] [squbs-akka.actor.default-dispatcher-26] [EventStream(akka://hcp-client)] Logger log1-Logging$DefaultLogger did not respond within Timeout(5000 milliseconds) to InitializeLogger(bus)
[error] error while starting up loggers
[error] akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.Logging$DefaultLogger] due to [akka.event.Logging$LoggerInitializationException: Logger log1-Logging$DefaultLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]]
[error]         at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$1.applyOrElse(Logging.scala:116)
[error]         at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$1.applyOrElse(Logging.scala:115)
[error]         at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
[error]         at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:215)
[error]         at scala.util.Try$.apply(Try.scala:191)
[error]         at scala.util.Failure.recover(Try.scala:215)
[error]         at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:115)
[error]         at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:110)
[error]         at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:728)
[error]         at scala.collection.Iterator$class.foreach(Iterator.scala:750)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1202)
[error]         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error]         at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:727)
[error]         at akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:110)
[error]         at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
[error]         at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:622)
[error]         at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:619)
[error]         at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:616)
[error]         at akka.actor.ActorSystemImpl._start(ActorSystem.scala:616)
[error]         at akka.actor.ActorSystemImpl.start(ActorSystem.scala:633)
[error]         at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
[error]         at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)
[error]         at com.paypal.vyasa.persistors.KafkaStorage.<init>(KafkaStorage.scala:40)
[error]         at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source)
[error]         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]         at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
[error]         at akka.util.Reflect$.instantiate(Reflect.scala:66)
[error]         at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
[error]         at akka.actor.Props.newActor(Props.scala:252)
[error]         at akka.actor.ActorCell.newActor(ActorCell.scala:552)
[error]         at akka.actor.ActorCell.create(ActorCell.scala:578)
[error]         at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
[error]         at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
[error]         at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
[error]         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
[error]         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
[error]         at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[error]         at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[error]         at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[error]         at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[error] Caused by: akka.event.Logging$LoggerInitializationException: Logger log1-Logging$DefaultLogger did not respond with LoggerInitialized, sent instead [TIMEOUT]
[error]         at akka.event.LoggingBus$class.akka$event$LoggingBus$$addLogger(Logging.scala:185)
[error]         at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:114)
[error]         at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:113)

I searched the web, but seems the error is unique. Any leads anyone ? Also any leads on how the performance can be inreased ? Total number of requests seems to throttles around 15 per second.

Cyprio

Do you have any SLF4J/logback warnings at the start? If so, correcting them may solve the problem. – Cardie

Answer 1

This error has been already mentioned in the web:

https://groups.google.com/forum/#!topic/akka-user/nng-bb2IZFA

It means that one of actors (log1-Logging) did not respond in time. You can increase the timeout using logger-startup-timeout setting:

 # Loggers are created and registered synchronously during ActorSystem
 # start-up, and since they are actors, this timeout is used to bound the
 # waiting time
 logger-startup-timeout = 5s

The possible reason why it may happen is executor service (pool) configuration:

• maybe you chose too low (or high) pool capacity
• maybe you're using pool, which may be influenced by thread starvation problem - any pool which doesn't support managed blocking (including regular built-in fixed and dynamic pools). This can be resolved by using fork-join-pool (which used by default in akka)

Looselimbed

Answer 2

This is probably due to LOGBACK-1221, which was fixed in Logback 1.1.10. If you're using an earlier version of Logback, the solution is simply to update to 1.1.10 or later.

Lament

For me it was 2 different version of logback-classic in the classpath. – Cardie