sparklyr write data to hdfs or hive

Question

I tried using sparklyr to write data to hdfs or hive , but was unable to find a way . Is it even possible to write a R dataframe to hdfs or hive using sparklyr ? Please note , my R and hadoop are running on two different servers , thus I need a way to write to a remote hdfs from R .

Regards Rahul

Chitkara

Have you tried to run Spark in yarn mode? This post might be helpful. – Elurd

Answer 1

Writing Spark table to hive using Sparklyr:

iris_spark_table <- copy_to(sc, iris, overwrite = TRUE)
sdf_copy_to(sc, iris_spark_table)
DBI::dbGetQuery(sc, "create table iris_hive as SELECT * FROM iris_spark_table")

Cristycriswell

thanks for sharing. this loads the data into hive's default database. do you know how to change the hive database for which to export the file? – Pavilion

@Pavilion You can use syntax database.table in the SQL passed to DBI. – Play

Answer 2

As of latest sparklyr you can use spark_write_table. pass in the format database.table_name to specify a database

iris_spark_table <- copy_to(sc, iris, overwrite = TRUE)
spark_write_table(
  iris_spark_table, 
  name = 'my_database.iris_hive ', 
  mode = 'overwrite'
)

Also see this SO post here where i got some input on more options

Limemann

Answer 3

You can use sdf_copy_to to copy a dataframe into Spark, lets say tempTable. Then use DBI::dbGetQuery(sc, "INSERT INTO TABLE MyHiveTable SELECT * FROM tempTable") to insert the dataframe records in a hive table.

Nullifidian