How to save a file on the cluster

Question

I'm connected to the cluster using ssh and I send the program to the cluster using

spark-submit --master yarn myProgram.py

I want to save the result in a text file and I tried using the following lines:

counts.write.json("hdfs://home/myDir/text_file.txt")
counts.write.csv("hdfs://home/myDir/text_file.csv")

However, none of them work. The program finishes and I cannot find the text file in myDir. Do you have any idea how can I do this?

Also, is there a way to write directly to my local machine?

EDIT: I found out that home directory doesn't exist so now I save the result as: counts.write.json("hdfs:///user/username/text_file.txt") But this creates a directory named text_file.txt and inside I have a lot of files with partial results inside. But I want one file with the final result inside. Any ideas how I can do this ?

Mitigate

Can you please show the output of hdfs dfs -ls hdfs://home/myDir? – Impetus

Also, if Spark uses HDFS as the default file system, you only need /home/myDir to write to – Impetus

-ls: java.net.UnknownHostException: home so I guess this folder doesn't exist. Usually when I what to save the file in with directory should I put it ? – Mitigate

You can place it anywhere... HDFS is empty by default. But /home is Linux user directory.... In HDFS, it's /user. – Impetus

UnknownHostException is because your path is wrong. It should be hdfs:///home/myDir, or better remove hdfs:// from everywhere, as mentioned – Impetus

@cricket_007 I understand now home directory doesn't exist but I can save it inside /user/username. But can I save it as a file instead of as a directory ? – Mitigate

Answer 1

Spark will save the results in multiple files since the computation is distributed. Therefore writing:

counts.write.csv("hdfs://home/myDir/text_file.csv")

means to save the data on each partition as a separate file in the folder text_file.csv. If you want the data saved as a single file, use coalesce(1) first:

counts.coalesce(1).write.csv("hdfs://home/myDir/text_file.csv")

This will put all the data into a single partition and the number of saved files will thus be 1. However, this could be a bad idea if you have a lot of data. If the data is very small then using collect() is an alternative. This will put all data onto the driver machine as an array, which can then be saved as a single file.

Hydrophilous

You could use : counts.repartition(1).write.csv("hdfs://home/myDir/text_file.csv"). But please note the repartition algorithm does a full shuffle of the data and creates equal sized partitions of data. coalesce combines existing partitions to avoid a full shuffle. The repartition method can be used to either increase or decrease the number of partitions in a DataFrame. However, the coalesce algorithm obviously cannot increate the number of partitions. – Indentation

Answer 2

You can concatenate your results into one file from the command line:

hadoop fs -cat hdfs:///user/username/text_file.txt/* > path/to/local/file.txt

This should be faster than using coalesce - in my experience all collect() type operations are slow because all of the data is funneled through the master node. Furthermore, you can run into troubles with collect() if your data exceeds the memory on your master node.

However, a potential pitfall with this approach is that you will have to explicitly remove the files from a previous run (since the current run may not produce exactly the same number of files). There may be a flag to do this with each run, but I am not sure.

To remove:

hadoop fs -rm -r hdfs:///user/username/text_file.txt/*

Crymotherapy

Answer 3

Do you get any error? Maybe you can check if you have the correct permissions to write/read from that folder.

Also think that Spark by default will create a folder called text_file.txt with some files inside, depending on the number of partitions that you have.

If you want to write in your local machine you can specify the path with file:///home/myDir/text_file.txt. If you use a path like /user/hdfs/... by default is wrote in HDFS

Explication

Can i check from the terminal if i have permission to write ? I am on Mac – Mitigate

You can use hdfs dfs -ls /home/myDir to see permissions and the owner of the folder, and also check which user are you using when running spark-submit. Maybe you could try to user /user/spark/... as a folder instead of /home. Home folder doesn't exists by default in HDFS. – Nodose

Do you how can I write the whole result in one txt file ? Because as you said it creates a directory and inside I have the partial results. But I want only one file with the final result inside. – Mitigate

Response from Shaido is that you need to create only one file. In any case, Spark will create a folder called text_file.csv with only 1 file inside. – Nodose

Answer 4

To have a unique file (not named as you want) you need .repartition(1),look here, piped to your RDD. I suppose that your hdfs path is wrong. In Spark HDFS for text file is the default and in Hadoop (by default) there is not a home dir in root dir, unless you have created it before. If you want a csv/txt file (with this extention) the only way to write it, is without RDD or DF functions, but using the usual libraries of python csv and io, after you have collected, with .collect(), your RDD in a martix (dataset has not be huge).

If you want to write directly on your filesystem (and not on HDFS) use

counts.write.csv("file:///home/myDir/text_file.csv")

But this won't write a single file with csv extension. It will create a folder with the part-m-0000n of the n partitions of your dataset.

Facile