Why spark executor cores are not equal with active tasks in spark web UI？

Question

I'm using Spark 2.3 thrift server for Ad-hoc Sql queries. My spark parameters are set as below in spark-defaults.conf file:

spark.executor.memory 24G
spark.executor.cores 40
spark.executor.instances 3

However when I checked the spark web ui,the spark cores were not equal with active tasks as the picture blow shows:

How could the active task nums bigger than the cores allocated? Any ideas? Thanks!

Discrepant

Task is smallest unit or processing and generally many tasks run on a single core (it is advisable as well to improve resource util and performance). – Livery

@NachiketKate Thanks for your reply. But why the num of active tasks always 41 not 42 or 43, when the load is heavy ? – Discrepant

Number of tasks are dependent on spark actions, parallelism settings, repartitions etc. You can also see what each task is doing in Spark UI and that will help you understand the task count. – Livery

Answer 1

I've seen the same thing. I'm not 100% sure, but I believe it is a race condition between the Task threadpool on the executor and the metrics reporting code.

If you click on the thread dump you will see the proper number of threads. However if try it 50 times (with a little luck) you will see an extra task thread is just sitting there in the TIMED_WAITING state.

Interplanetary