Spark Memory Overhead

Question

Spark memory overhead related question asked multiple times in SO, I went through most of them. However, after going through multiple blogs, I got confused.

Below are the questions I have

• whether memory overhead is part of the executor memory or it's separate? As few of the blogs are saying memory overhead is part of the executor memory and others are saying executor memory + memory overhead(is that mean memory overhead is not part of the executor memory)?
• Memory overhead and off-heap over are the same?
• What happens if I didn't mention overhead as part of the spark-submit, will it take default 18.75 or it won't?
• Will there be any side effects if we give more memory overhead than the default value?

https://docs.qubole.com/en/latest/user-guide/engines/spark/defaults-executors.html https://spoddutur.github.io/spark-notes/distribution_of_executors_cores_and_memory_for_spark_application.html

Below is the case I want to understand. I have 5 nodes with each node 16 vcores and 128GB Memory(out of which 120 is usable), now I want to submit spark application, below is the conf, I'm thinking

Total Cores 16 * 5 = 80
Total Memory 120 * 5 = 600GB

case 1: Memory Overhead part of the executor memory

spark.executor.memory=32G
spark.executor.cores=5
spark.executor.instances=14 (1 for AM)
spark.executor.memoryOverhead=8G ( giving more than 18.75% which is default)
spark.driver.memoryOverhead=8G
spark.driver.cores=5

Case 2: Memory Overhead not part of the executor memory

spark.executor.memory=28G
spark.executor.cores=5
spark.executor.instances=14 (1 for AM)
spark.executor.memoryOverhead=6G ( giving more than 18.75% which is default)
spark.driver.memoryOverhead=6G
spark.driver.cores=5

As per the below video, I'm trying to use 85% of the node i.e. around 100GB out of 120GB, not sure if we can use more than that.

https://www.youtube.com/watch?v=ph_2xwVjCGs&list=PLdqfPU6gm4b9bJEb7crUwdkpprPLseCOB&index=8&t=1281s (4:12)

Shrewmouse

whether memory overhead is part of the executor memory or it's separate? yes ... in resource manager launches containers in order to execute executors inside that. so basically executor memory + memory overhead = container memory ..... spark have breakage for executor memory in to application memory and cache memory – Ossie

and executor memory overhead includes offheap memory and buffers and memory for running container-specific threads. – Ossie

What happens if I didn't mention overhead as part of the spark-submit, will it take default... The resource manager calculates memory overhead value by using default values if not mentioned explicitly. – Ossie

@kavetiraviteja, Can I use 100% of the node i.e. 120GB in the above case or 85% is recommended? any thoughts? Which config will be best(in general, may vary specific to application) for the above 5 nodes? – Shrewmouse

Total Cores 16 * 5 = 80 Total Memory 120 * 5 = 600GB ...... you should always keep aside cores and memory for OS which is running on that node and 1 core for nodemanager and 1 core for other daemons and 2 cores for OS to work optimally – Ossie

assuming 12*5 = 60 and total memory 116*5 = 580GB is what total resources available .. then you tune other parameters correspondingly... – Ossie

Is there any way to detect the cause of "memory overhead " issue? How can I know which part of the off-heap memory(etc. direct buffer) exhausted the off-heap memory? – Libre

Answer 1

To answer your question whether memory overhead is part of the executor memory or it's separate? Memory Overhead is not part of executor memory.

Resource manager launches containers in order to execute executors inside it. so basically executor memory + memory overhead = container memory ..... spark have break up for executor memory into application memory and cache memory.

Executor memory overhead mainly includes off-heap memory and nio buffers and memory for running container-specific threads(thread stacks). when you do not specify memory overhead, Resource manager calculates memory overhead value by using default values and launch containers accordingly.

It is always recommended to keep aside cores and memory for OS (which is 1 core for nodemanager and 1 core for other daemons and 2 cores for OS to work optimally)

You can change your calculation like below mentioned 12 * 5 = 60cores and total memory 116 * 5 = 580GB is what total resources available .. then you tune other parameters correspondingly.

Ossie

executor memory overhead does not include off-heap memory in 3.x. From documentation: "The maximum memory size of container to running executor is determined by the sum of spark.executor.memoryOverhead, spark.executor.memory, spark.memory.offHeap.size and spark.executor.pyspark.memory." – Dehaven

Answer 2

1. Memory overhead is not part of the executor memory. Executor memory is set by spark.executor.memory. Memory overhead is part of the container memory.
2. Both memory overhead and the amount of memory defined by spark.memory.offHeap.size are allocated outside the JVM Heap. The below image, sourced from the medium article mentioned below, clearly depicts the definition and usage of both.
3. There will be a default value for the amount memory overhead used:

max(executorMemory * spark.executor.memoryOverheadFactor, 384 MiB)

4. As this portion is allocated from the off-heap memory, there will not be any garbage collection. So we might have to be care in managing the memory.

Reference: https://medium.com/walmartglobaltech/decoding-memory-in-spark-parameters-that-are-often-confused-c11be7488a24

Germaun