Using Jackson 2.9.9 in java Spark

Question

I am trying to use the MLLIB library (java) but one of my dependencies uses Jackson 2.9.9. I noticed that a pull request was made such that the master branch's dependency is upgraded to this particular version. Now I wanted to use this master branch so I cloned and installed this package using ./dev/make-distribution.sh. but not sure which jar packages to copy to my project. I tried to copy all the 3.0.0-snaphots but I end up with a class file for scala.Cloneable not found in my java program.

I also did a ./build/mvn -DskipTests install which installs the SNAPSHOT dependencies under ~/.m2.

I added them to the gradle build file

    // https://mvnrepository.com/artifact/org.apache.spark/spark-core
    compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.0.0-SNAPSHOT'

    // https://mvnrepository.com/artifact/org.apache.spark/spark-mllib
    compile group: 'org.apache.spark', name: 'spark-mllib_2.12', version: '3.0.0-SNAPSHOT'

but upon execution under java 11, it complains about:

Exception in thread "main" java.lang.NoSuchFieldError: JAVA_9
    at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)

Since java 9 is not available anymore I am wondering how to further debug this issue. I assume at least java 9 is required but the field does no longer exists?

Spermatozoon

Answer 1

I notice that in the sourcecode StorageUtils.scala, it is pulling in apache common-lang3 for JAVA_9. Have you check whether apache common-lang3 is included in the classpath? It solved my problem by adding commons-lang3-3.9.jar to my classpath.

Indigoid

This would have been better as a comment rather than an answer – Lucianaluciano

why? @Indigoid suggested some possible solution that might work. – Offcolor

Spark 3 requires commons-lang3 >= 3.5. Adding this library (and making sure you get rid of older versions) fixed the problem for me when upgrading from Spark 2. – Refinery