How multiple executors are managed on the worker nodes with a Spark standalone cluster?

Question

Until now, I have only used Spark on a Hadoop cluster with YARN as the resource manager. In that type of cluster, I know exactly how many executors to run and how the resource management works. However, know that I am trying to use a Standalone Spark Cluster, I have got a little bit confused. Correct me where I am wrong.

From this article, by default, a worker node uses all the memory of the node minus 1 GB. But I understand that by using SPARK_WORKER_MEMORY, we can use lesser memory. For example, if the total memory of the node is 32 GB, but I specify 16 GB, Spark worker is not going to use anymore than 16 GB on that node?

But what about executors? Let us say if I want to run 2 executors per node, can I do that by specifying executor memory during spark-submit to be half of SPARK_WORKER_MEMORY, and if I want to run 4 executors per node, by specifying executor memory to be the quarter of SPARK_WORKER_MEMORY?

If so, besides executor memory, I would also have to specify executor cores correctly, I think. For example, if I want to run 4 executors on a worker, I would have to specify executor cores to be the quarter of SPARK_WORKER_CORES? What happens, if I specify a bigger number than that? I mean if I specify executor memory to be the quarter of SPARK_WORKER_MEMORY, but executor cores to be only half of SPARK_WORKER_CORES? Would I get 2 or 4 executors running on that node in that case?

Merovingian

Can you check if the acceptable answer still makes sense – Accuracy

Answer 1

So, I experimented with the Spark Standalone cluster myself a bit, and this is what I noticed.

1. My intuition that muliple executors can be run inside a worker, by tuning executor cores was indeed correct. Let us say, your worker has 16 cores. Now if you specify 8 cores for executors, Spark would run 2 executors per worker.

2. How many executors run inside a worker also depend upon the executor memory you specify. For example, if worker memory is 24 GB, and you want to run 2 executors per worker, you cannot specify executor memory to be more than 12 GB.

3. A worker's memory can be limited when starting a slave by specifing the value for optional parameter--memory or by changing the value of SPARK_WORKER_MEMORY. Same with the number of cores (--cores/SPARK_WORKER_CORES).

If you want to be able to run multiple jobs on the Standalone Spark cluster, you could use the spark.cores.max configuration property while doing spark-submit. For example, like this.

spark-submit <other parameters> --conf="spark.cores.max=16" <other parameters>

So, if your Standalone Spark Cluster allows 64 cores in total, and you give only 16 cores to your program, other Spark jobs could use the remaining 48 cores.

Merovingian

Look at my comment in the above, it doesn't work as expected and I can confirm the same from another stackoverflow post. – Accuracy

Thanks @Accuracy for pointing it out. I have updated my answer accordingly. – Merovingian

Answer 2

This is the best way to control number of executors, cores and memory in my experience.

• Cores: You can set total number of cores across all executors and number of cores per each executor

• Memory: Executor memory individually

  --total-executor-cores 12 --executor-cores 2 --executor-memory 6G

  This would give you 6 executors and 2 cores/6G per each executor, so in total you are looking at 12 Cores and 36G

• You can set driver memory using

  --driver-memory 2G

Accuracy

Who said you cannot specify the number of executors? Because I do it all the times using --num-executors with spark-submit. – Merovingian

I have tried --num-executors 6 and executors cores 3. We have a cluster with 6 nodes and 12 cores each, I ended up with 23 executors with 3 cores when I launched a job, doesn't look like it makes sense. – Accuracy

@Merovingian I can confirm the same from another post, #39399705 Check your facts – Accuracy