ClassNotFoundException scala.runtime.LambdaDeserialize when spark-submit

Question

I follow the Scala tutorial on https://spark.apache.org/docs/2.1.0/quick-start.html

My scala file

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "/data/README.md" // Should be some file on your system
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println(s"Lines with a: $numAs, Lines with b: $numBs")
    sc.stop()
  }
}

and build.sbt

name := "Simple Project"

version := "1.0"

scalaVersion := "2.12.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0" 

I ran sbt package sucessfully (already delete everything except scala source code and build.sbt then run sbt package again)

[info] Loading project definition from /home/cpu11453local/workspace/testspark_scala/project
[info] Loading settings from build.sbt ...
[info] Set current project to Simple Project (in build file:/home/my_name/workspace/testspark_scala/)
[info] Packaging /home/my_name/workspace/testspark_scala/target/scala-2.12/simple-project_2.12-1.0.jar ...
[info] Done packaging.
[success] Total time: 1 s, completed Nov 8, 2017 12:15:24 PM

However, when I run spark submit

$SPARK_HOME/bin/spark-submit --class "SimpleApp" --master local[4] simple-project_2.12-1.0.jar 

I got error

java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize

Full spark-submit output on gist

Nanette

You are using incompatible Scala and Spark versions, see e.g. #43883825 – Misapprehension

Change scalaVersion := "2.12.4" to scalaVersion := "2.11.8" and "spark-core_2.10" to "spark-core_2.11". – Caslon

BTW, if you've just started working with Spark, use spark.apache.org/docs/latest/sql-programming-guide.html instead. Please. – Caslon

Answer 1

as @Alexey said, change Scala version to 2.11 fixed the problem.

build.sbt

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.11"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0" 

Note that Scala version MUST MATCH with Spark. Look at the artifactId, spark-core_2.11 mean it was compatible with scala 2.11 (No backward or forward compatible)

Nanette

Answer 2

Following is the build.sbt entries for the latest Spark 2.4.1 release sample shown in Spark/Scala online guide:

name := "SimpleApp" 
version := "1.0"
scalaVersion := "2.12.8"
libraryDependencies += "org.apache.spark"  %% "spark-sql" % "2.4.1"

Though everything works fine inside IntelliJ IDE, the application still fails with the following exception,

Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize

after creating the package with 'sbt package' command and running the spark-submit from the command line as the following;

spark-submit -v --class SimpleApp --master local[*] target\scala-2.12\simpleapp_2.12-1.0.jar

Platinotype

if it is a question, please create as a new question – Nanette

Yes, this is a new problem with latest Spark version 2.4.1 using Scala 2.12.8 – Platinotype

This issue has been resolved after new Spark 2.4.2 release today – Platinotype

@diopek, no, it isn't. I'm experiencing the same issue trying to spark-submit a sample app built with Spark 2.4.3 and Scala 2.12.8. – Carabao

@Carabao The Spark 2.4.3 pre-built distribution went BACK to being built with Scala 2.11.12. Here's the complete list: Spark 1.x built with Scala 2.10. Spark 2.0.0 - Spark 2.4.1 built with Scala 2.11. Spark 2.4.2 is the ONLY version built with Scala 2.12 at the moment. Spark 2.4.3 is back to Scala 2.11. – Vermination

Answer 3

I have similar issue while following the instructions provided at https://spark.apache.org/docs/2.4.3/quick-start.html

My setup details: Spark version: 2.4.3 Scala version: 2.12.8

However, when i changed my sbt file to below configuration everything worked fine.(both compilation and running the application jar)

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.11"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.3"

It looks like spark 2.4.3 is compatible with 2.11.11 Scala version only. While compiling the sample project sbt has downloaded the Scala 2.11 library from "https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.11"

Pirnot

Answer 4

There is definitely some confusion regarding the Scala version to be used for Spark 2.4.3. As of today (Nov 25, 2019) the doc home page for spark 2.4.3 states:

Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.3 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

Note that support for Java 7, Python 2.6 and old Hadoop versions before 2.6.5 were removed as of Spark 2.2.0. Support for Scala 2.10 was removed as of 2.3.0. Support for Scala 2.11 is deprecated as of Spark 2.4.1 and will be removed in Spark 3.0.

Accordingly, the Scala version is supposed to be 2.12.

Chester

Answer 5

I used sdkman to install scala and spark

I solved this[3] by:

• finding the versions I installed[1]
• updating build.sbt[2]

[1]

spark/2.4.7/hello_world via ☕ v1.8.0 via 🆂 vsuch on ☁️  (us-west-2) took 11s
❯ scala -version
cat: /Users/lgeoff/.sdkman/candidates/java/current/release: No such file or directory
Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

spark/2.4.7/hello_world via ☕ v1.8.0 via 🆂 vsuch on ☁️  (us-west-2)
❯

spark/2.4.7/hello_world via ☕ v1.8.0 via 🆂 vsuch on ☁️  (us-west-2)
❯ sdk list spark
================================================================================
Available Spark Versions
================================================================================
     3.2.0               2.3.2
     3.1.2               2.3.1
     3.1.1               2.3.0
     3.0.2               2.2.1
     3.0.1               2.2.0
     3.0.0               2.1.3
 > * 2.4.7               2.1.2
     2.4.6               2.1.1
     2.4.5               2.0.2
     2.4.4               1.6.3
     2.4.3               1.5.2
     2.4.2               1.4.1
     2.4.1
     2.4.0
     2.3.3

================================================================================
+ - local version
* - installed
> - currently in use
================================================================================
spark/2.4.7/hello_world via ☕ v1.8.0 via 🆂 vsuch on ☁️  (us-west-2)
❯

[2]

spark/2.4.7/hello_world via ☕ v1.8.0 via 🆂 vsuch on ☁️  (us-west-2) took 8s
❯ cat build.sbt
name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.12"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.7"

spark/2.4.7/hello_world via ☕ v1.8.0 via 🆂 vsuch on ☁️  (us-west-2)
❯

[3]

❯ spark-submit \
  --class "SimpleApp" \
  --master local[4] \
  target/scala-2.11/simple-project_2.11-1.0.jar

...

Lines with a: 61, Lines with b: 30

...

❯

Cerda