How to detect scala executioncontext exhaustion?

Question

I'm having problems with my Playframework application not being responsive from time to time and I would like to detect this at runtime + log information on what is currently running on the exhausted execution context.

What would the best strategy for implementing this be? I thought about posting small runnables to the execution contexts and if they don't get executed in time I would log a warning. This max wait time should of course be configurable. Eg the main web execution context should never be blocked for more than 1 sec but a background db execution context might allow 30 sec of blocking.

Somebody must have done this before?

related info: http://www.playframework.com/documentation/2.2.x/ThreadPools

Selda

would you like to detect if the CPU/memory is being used to its limits? – Anthesis

Is New Relic a possible option for you? – Waac

This has nothing to do with the os/cpu/memory, it's about setting up the correct akka execution contexts and having tools that help you detect performance degradation because of using the wrong execution contexts in the code. – Selda

Answer 1

That is pretty hard question to answer.

Configuration of dispatchers really depends on the type of work your actors are doing.

Probably you should look at the actors that spawn futures to do their work. It may be a good idea to predefine execution context in the configuration file and use them like this:

implicit val ec : ExecutionContext = context.system.dispatchers.lookup("someDispatcher")

those actors are likely to cause the starvation effect.

The effect you want to achieve is to make sure messages are processed quickly and long running task does not affect them so separation is the key thing here.

A good tool for monitoring your application is the Typesafe Console. You can look at the dispatcher there and see that the latency in handling messages increase.

Another concern is to identify actors that make a high risk work like network I/O. If something happens to the thread making in not available again in the pool it will create problems.

It is very likely that without experimenting with the threadpool size you won't know what is the best setting for you.

Most of those advises I wrote I know from the book Effective Akka by Jamie Allen and they seem to be working quite well for me. There is a section in this book Fixing Starvation. You may want to look at it closer.

Becht

thanks for the response, but let's say that you have an intern working on the project and he does not know about the rules of what can be run on what dispatcher. He might put a long-blocking operation on the web dispatcher. I would like the system to tell him that he is doing it/something wrong... – Selda

I believe achieving this automatically would be pretty hard to get. To be honest I don't have a good idea how to help the 'poor' intern. – Becht

Something that you may consider is adding stack of traits to your actor with some periodical metrics e.g. number of messages processed and time spent in the receive block. Then gather them and expose them to analysis. – Becht

the problem is that we don't/don't want to control the play internals, anyway, I'll try to come up with something and share it afterwards – Selda

Great. If you have something please post it here. It may be very beneficial for others. – Becht

Any updates on this topic? I'd be really interested if/how such monitoring can be achieved using logs. – Boliviano