Running custom Java class in PySpark

Question

I'm trying to run a custom HDFS reader class in PySpark. This class is written in Java and I need to access it from PySpark, either from the shell or with spark-submit.

In PySpark, I retrieve the JavaGateway from the SparkContext (sc._gateway).

Say I have a class:

package org.foo.module

public class Foo {

    public int fooMethod() {
        return 1;
    }

}

I've tried to package it into a jar and pass it with the --jar option to pyspark and then running:

from py4j.java_gateway import java_import

jvm = sc._gateway.jvm
java_import(jvm, "org.foo.module.*")

foo = jvm.org.foo.module.Foo()

But I get the error:

Py4JError: Trying to call a package.

Can someone help with this? Thanks.

Shoreline

Answer 1

In PySpark try the following

from py4j.java_gateway import java_import
java_import(sc._gateway.jvm,"org.foo.module.Foo")

func = sc._gateway.jvm.Foo()
func.fooMethod()

Make sure that you have compiled your Java code into a runnable jar and submit the spark job like so

spark-submit --driver-class-path "name_of_your_jar_file.jar" --jars "name_of_your_jar_file.jar" name_of_your_python_file.py

Darcidarcia

Also, remember if you are adding multiple jars make sure to use classpath syntax for --driver-class-path and comma separation --jars. – Firstborn

Adding --driver-class-path causes tons of issues for me within AWS / EMR. Just adding --jars was enough for me and fixed tons of issues I saw when also adding the same jar to --driver-class-path (which broke Hive and S3 access, to name a few). – Silicone

Answer 2

Problem you've described usually indicates that org.foo.module is not on the driver CLASSPATH. One possible solution is to use spark.driver.extraClassPath to add your jar file. It can be for example set in conf/spark-defaults.conf or provided as a command line parameter.

On a side note:

• if class you use is a custom input format there should be no need for using Py4j gateway whatsoever. You can simply use SparkContext.hadoop* / SparkContext.newAPIHadoop* methods.

• using java_import(jvm, "org.foo.module.*") looks like a bad idea. Generally speaking you should avoid unnecessary imports on JVM. It is not public for a reason and you really don't want to mess with that. Especially when you access in a way which make this import completely obsolete. So drop java_import and stick with jvm.org.foo.module.Foo().

Coopersmith

Using the classpath option actually worked and I can use the classes in the Spark driver. However, when I try to use them inside transformations I get different kind of errors. The option of SparkContext.hadoop* doesn't fit my use case. I want to parallelize a list of paths and then make a transformation that reads those files. – Shoreline

Inside transformations? It is not possible (or at least not using this approach). – Coopersmith

You can also add it to the classpath by adding it as a cmd-line param with: --driver-class-path if you don't want to change your config files – Reid

Answer 3

If you run PySpark locally in IDE (PyCharm, etc.), to use custom classes in a jar, you can put the jar into $SPARK_HOME/jars, it will be added to class path to run Spark, check code snippet in $SPARK_HOME/bin/spark-class2.cmd for details.

Calmas

Answer 4

Rather than --jars you should use --packages to import packages into your spark-submit action.

Advancement

This is not always correct. --packages searches for Maven packages. If a user is attempting to load their own JAR that is not in a Maven repo, --jars is correct. – Pentadactyl