Starting SparkR session using external config file

Question

I have an RStudio driver instance which is connected to a Spark Cluster. I wanted to know if there is any way to actually connect to Spark cluster from RStudio using an external configuration file which can specify the number of executors, memory and other spark parameters. I know we can do it using the below command

sparkR.session(sparkConfig = list(spark.cores.max='2',spark.executor.memory = '8g'))

I am specifically looking for a method which takes spark parameters from an external file to start the sparkR session.

Simonsen

Answer 1

Spark uses standardized configuration layout with spark-defaults.conf used for specifying configuration option. This file should be located in one of the following directories:

• SPARK_HOME/conf
• SPARK_CONF_DIR

All you have to do is to configure SPARK_HOME or SPARK_CONF_DIR environment variables and put configuration there.

Each Spark installation comes with template files you can use as an inspiration.

Mimosaceous

What I actually need is that I should be able to pass an external spark configuration file through RStudio while connecting to spark so that it does not use the default configurations from spark-defaults.conf file – Simonsen