Akka - during load testing, forkjoinpool.scan at 20% of cpu time

Question

We're making some good progress in load testing and scaling an akka application but we're seeing scala.concurrent.forkjoin.ForkJoinPool.scan() coming up as the second highest hotspot around 20% of self time in visualvm. The Self time (CPU) column says only a fraction of that (less than 1% of the value of the self time column).

I suspect this means blocking or context switching are potentially problematic but I'm not too sure - can any one give insight? If it's context switching I'm guessing tuning dispatcher throughput to a higher number may net us gains, otherwise if it's caused by blocking we'll need to read through the code some more.

Any insight greatly appreciated.

Maugham

I would read through this thread as it provides some good insight into why you might be seeing a lot of busy wait time with the FJP. You might indeed have some blocking code which is slowing things down causing the idle FJP threads to be scanning for work more than expected. groups.google.com/forum/m/#!topic/akka-user/6HKTvw4yBnU – Schalles

Thanks - that thread is what sparked me off thinking that there must be some parked threads - that and the wall clock time vs the actual cpu used. This is a helpful affirmation +1 – Maugham

Answer 1

The scan method is also where inactive threads in the ForkJoinPool "parks" when they can't find work.

Having 20% self time and 1% of that as self CPU time makes me think that you have inactive threads in the ForkJoinPool that have parked in there.

I'm assuming that you are using VisualVM in Sampling mode. By default VisualVM filters out things in the standard library. See this answer how to change the settings and get a better picture of what's going on. https://mcmap.net/q/1165517/-why-is-this-method-a-hot-spot

Ranita

Okay that's awesome - thank you muchly. I'll try and report back. – Maugham

Yes indeed - uncovering the sun.* showed sun.misc.Unsafe.park was eating the cpu. – Maugham