spark: java.io.IOException: No space left on device [again!]

Question

I am getting the java.io.IOException: No space left on device that occurs after running a simple query in sparklyr. I use both last versions of Spark (2.1.1) and Sparklyr

df_new <-spark_read_parquet(sc, "/mypath/parquet_*", name = "df_new", memory = FALSE)

myquery <- df_new %>% group_by(text) %>% summarize(mycount = n()) %>% 
  arrange(desc(mycount)) %>% head(10)

#this FAILS
get_result <- collect(myquery)

I do have set both

• spark.local.dir <- "/mypath/"
• spark.worker.dir <- "/mypath/"

using the usual

config <- spark_config()

config$`spark.executor.memory` <- "100GB"
config$`spark.executor.cores` <- "3"
config$`spark.local.dir` <- "/mypath/"
config$`spark.worker.dir` <- "mypath/"
config$`spark.cores.max`<- "2000"
config$`spark.default.parallelism`<- "4"
config$`spark.total-executor-cores`<- "80"
config$`sparklyr.shell.driver-memory` <- "100G"
config$`sparklyr.shell.executor-memory` <- "100G"
config$`spark.yarn.executor.memoryOverhead` <- "100G"
config$`sparklyr.shell.num-executors` <- "90"
config$`spark.memory.fraction` <- "0.2"

  Sys.setenv(SPARK_HOME="mysparkpath")
  sc <- spark_connect(master = "spark://mynode", config = config)

where mypath has more than 5TB of disk space (I can see these options in the Environment tab). I tried a similar command in Pyspark and it failed the same way (same error).

By looking at the Stages tab in Spark, I see that the error occurs when shuffle write is about 60 GB. (input is about 200GB). This is puzzling given that I have plenty of space available. I have have looked at the other SO solutions already...

The cluster job is started with magpie https://github.com/LLNL/magpie/blob/master/submission-scripts/script-sbatch-srun/magpie.sbatch-srun-spark

Every time I start a Spark job, I see a directory called spark-abcd-random_numbers in my /mypath folder. but the size of the files in there is very small (nowhere near the 60GB shuffle write)

• there are about 40 parquet files. each is 700K (original csv files were 100GB) They contain strings essentially.
• cluster is 10 nodes, each has 120GB RAM and 20 cores.

What is the problem here? Thanks!!

Gnosticize

can you provide the spark command you re using to show what is the master and deploy mode. that could help a lot in your case – Illogic

@Illogic I am using slurm actually. I can see my master node by typing squeue -u myname. is this what you mean? – Send

I was looking for the spark-submit command that you issued, but I ve see you re using some other method to start you application. so the question become : how did you manage to set both spark.local.dir and spark.worker.dir properties ? do you have access to spark config files of the cluster? – Illogic

I set them as arguments in sparklyr (see updated question). In the Spark UI Environment tab, I can see these options appear. So I guess they are taken into account? – Send

@Illogic does that help? – Send

can you check with watch "df -u" while your job is running which disk fills? is it root volume? Then I'd check what kind of files are filling it up – Skiba

@IgorBerman thanks ! df-u prints gazillion of disks. Do I need to spam the command or the disk usage refreshes by itself on the screen? – Send

@Noobie, the point is to find which disk is filling up and why. watch command usually executes sub-command once in a while(every 2 secs by default, you can control it...) – Skiba

@IgorBerman the -u argument does not work for me. It that the right command?. you mean running df and du? – Send

@IgorBerman by spamming df -H I first saw /tmp/ filling up to 80% and then no further actitivy on this disk or on other disks visibles by hitting df -H Meanwhile shuffle write was going from 10GB to almost 80GB then the job crashed on the no disk space available. After the crash, no disk is actually 100% full. how can that be? – Send

@IgorBerman any ideas? thanks!! – Send

@Noobie, the only idea that I have - maybe you are using sparkR shell or something(? I haven't used it) and you put your application inside this shell, so what is really works is configuration of shell and not spark config that you are providing...you've already got advice of restarting machine, but if you have spark-slave process(CoarseGrained something, try to find with ps -ef) - you can restart it first. We've talked about dir - are you using spark local context? is it the only machine you are using? – Skiba

@IgorBerman I start the Spark job using a magpie script. Please see updated question. I set the parameters there basically. Do you see something that might help? – Send

@Noobie, have you tried to play with following setting in script: export MAGPIE_LOCAL_DIR="/tmp/${USER}/magpie" ? also export SPARK_LOCAL_DIR="/tmp/${USER}/spark" – Skiba

@Noobie Have you tried to modify the magpie script to provide the java options for the spark job? – Illogic

Answer 1

I ve had this problem multiple times before. The reason behind is the temporary files. most of servers have a very small size partition for /tmp/ which is the default temporary directory for spark.
Usually, I used to change that by setting that in spark-submit command as the following:

$spark-submit --master local[*] --conf "spark.driver.extraJavaOptions=-Djava.io.tmpdir=/mypath/" ....

In your case, I think that you can provide that to the configuration in R as following (I have not tested that but that should work):

config$`spark.driver.extraJavaOptions` <- "-Djava.io.tmpdir=/mypath/"
config$`spark.executor.extraJavaOptions ` <- "-Djava.io.tmpdir=/mypath/"

Notice that you have to change that for the driver and executors since you're using Spark standalone master (as I can see in your question)

I hope that will help

Illogic

still getting the error.... the option appear in the Environment tab as spark.executor.extraJavaOptions -Djava.io.tmpdir=/mypath. is that correct? – Send

and what is the difference between the config$spark.worker.dir` <- "mypath/"` I was using? thx! – Send

the options are correct, they are both for the driver and the executors so you should also have spark.driver.extraJavaOptions -Djava.io.tmpdir=/mypath . The difference that working dir is a spark options where setting the java.io.tmpdir managing the java process of spark and will override all the properties in spark – Illogic

damn, do you see another way to make progress here? am i screwing up with the other cluster options? data is huge and cannot fit into ram – Send

also does it matter how the path is written? say \\mypath\myfolder or \\mypath\myfolder\ – Send

I was looking on the sparklyr docs, there is no reason that the configuration were not taken into consideration. I ll try to find out how to sort out that problem – Illogic

thanks I ll update the questions with more details about the data – Send

thanks man! my problem is not solved but i m sure the solutions on this page will help other people out! – Send

I ve just seen your update, since you re using magpie script there is a chance to add those config on the script: github.com/LLNL/magpie/blob/master/submission-scripts/… . here you can add export SPARK_JOB_JAVA_OPTS="-Djava.io.tmpdir=/mypath/" . do not forget to uncomment this line by removing the heading # – Illogic

Answer 2

change following settings in your magpie script

export MAGPIE_LOCAL_DIR="/tmp/${USER}/magpie" 
export SPARK_LOCAL_DIR="/tmp/${USER}/spark"

to have mypath prefix and not /tmp

Skiba

do I need to keep the user/magpie stuff? – Send

it's your decision, but I'd put it as is, so there will be clear differentiation – Skiba

Answer 3

Once you set the parameter, you can see the new value of spark.local.dir in Spark environment UI. But it doesn't reflect.

Even I faced the similar problem. After setting this parameter, I restarted the machines and then started working.

Mittel

what do you mean you restarted the machines? I cannot restart the machines every time I run a job – Send

set these parameters in spark-defaults.conf file and restart the servers. Then no need to pass these parameters from outside. – Mittel

If that is the case, try to set in gateway machine and check once. – Mittel

Answer 4

Since you need to set this when the JVM is launched via spark-submit, you need to use the sparklyr java-options, e.g.

config$`sparklyr.shell.driver-java-options` <- "-Djava.io.tmpdir=/mypath"

Assiduity

Answer 5

I had this very problem this week on a Standalone mode cluster and after trying different things, like some of the recommendations in this thread, it ended up being a sub folder called "work" inside the Spark home folder grew unchecked for a while thus filling up the worker's hhd

Noway