How to append a resource jar for spark-submit?

Question

My spark application depends on adam_2.11-0.20.0.jar, every time I have to package my application with adam_2.11-0.20.0.jar as a fat jar to submit to spark.

for example, my fat jar is myApp1-adam_2.11-0.20.0.jar,

It's ok to submit as following

spark-submit --class com.ano.adam.AnnoSp myApp1-adam_2.11-0.20.0.jar

It reported Exception in

thread "main" java.lang.NoClassDefFoundError:

org/bdgenomics/adam/rdd using --jars

spark-submit --class com.ano.adam.AnnoSp myApp1.jar --jars adam_2.11-0.20.0.jar

My question is how to submit using 2 separate jars without package them together

spark-submit --class com.ano.adam.AnnoSp myApp1.jar adam_2.11-0.20.0.jar

Douce

Answer 1

Add all jars in one folder and then do like below...

Option 1 :

I think Better way of doing this is

$SPARK_HOME/bin/spark-submit \
--driver-class-path  $(echo /usr/local/share/build/libs/*.jar | tr ' ' ',') \
--jars $(echo /usr/local/share/build/libs/*.jar | tr ' ' ',') 

in this approach, you wont miss any jar by mistake in the classpath hence no warning should come.

Option 2 see my anwer:

spark-submit-jars-arguments-wants-comma-list-how-to-declare-a-directory

Option 3 : If you want to do programmatic submit by adding jars through API its possible.Here Im not going to details of it.

Villon

was it helpful? your resource jar as well you can copy to that folder – Villon

spark-submit --driver-class-path adam_2.11-0.20.0.jar --class com.ano.adam.AnnoSp myApp1.jar works for spark2.0.1 – Douce