I'm connected to the cluster using ssh and I send the program to the cluster using spark-submit --master yarn myProgram.py I want to save the result in a text file and I tried using the followi...

I have recently found a way to use logback instead of log4j in Apache Spark (both for local use and spark-submit). However, there is last piece missing. The issue is that Spark tries very hard not ...

build.sbt lazy val commonSettings = Seq( organization := "com.me", version := "0.1.0", scalaVersion := "2.11.0" ) lazy val counter = (project in file("counter")). settings(commonSettings:_*) ...

When I submit spark job using spark-submit with master yarn and deploy-mode cluster, it doesn't print/return any applicationId and once job is completed I have to manually check MapReduce jobHistor...

I am currently running spark 2.1.0. I have worked most of the time in PYSPARK shell, but I need to spark-submit a python file(similar to spark-submit jar in java) . How do you do that in python?

We want to set the aws parameters that from code would be done via the SparkContext: sc.hadoopConfiguration.set("fs.s3a.access.key", vault.user) sc.hadoopConfiguration.set("fs.s3a.secret.key", vau...

I am beginner to Spark and I am running my application to read 14KB data from text filed, do some transformations and actions(collect, collectAsMap) and save data to Database I am running it local...

My spark application depends on adam_2.11-0.20.0.jar, every time I have to package my application with adam_2.11-0.20.0.jar as a fat jar to submit to spark. for example, my fat jar is myApp1-adam_...

In my Application, i need to connect to the database so i need to pass IP address and database name when application is submitted. I submit the application as follows: : ./spark-submit --class cl...