Mllib dependency error

Question

I'm trying to build a very simple scala standalone app using the Mllib, but I get the following error when trying to bulid the program:

Object Mllib is not a member of package org.apache.spark

Then, I realized that I have to add Mllib as dependency as follow :

version := "1"
scalaVersion :="2.10.4"

libraryDependencies ++= Seq(
"org.apache.spark"  %% "spark-core"              % "1.1.0",
"org.apache.spark"  %% "spark-mllib"             % "1.1.0"
)

But, here I got an error that says :

unresolved dependency spark-core_2.10.4;1.1.1 : not found

so I had to modify it to

"org.apache.spark" % "spark-core_2.10" % "1.1.1",

But there is still an error that says :

unresolved dependency spark-mllib;1.1.1 : not found

Anyone knows how to add dependency of Mllib in .sbt file?

Laundry

Something weird seems to be happening with your sbt and scala binary compatible versions, but doing it explictly the same way as you did with spark-core: "org.apache.spark" % "spark-mllib_2.10" % "1.1.1" should work. (I build all my projects with Maven so I don't know SBT especially well). – Nitrite

Answer 1

As @lmm pointed out, you can instead include the libraries as:

libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.10" % "1.1.0", "org.apache.spark" % "spark-mllib_2.10" % "1.1.0" )

In sbt %% includes the scala version, and you are building with scala version 2.10.4 whereas the Spark artifacts are published against 2.10 in general.

It should be noted that if you are going to make an assembly jar to deploy your application you may wish to mark spark-core as provided e.g.

libraryDependencies ++= Seq( "org.apache.spark" % "spark-core_2.10" % "1.1.0" % "provided", "org.apache.spark" % "spark-mllib_2.10" % "1.1.0" )

Since the spark-core package will be in the path on executor anyways.

Chrismatory

Answer 2

Here is another way to add the dependency to your build.sbt file if you're using the Databricks sbt-spark-package plugin:

sparkComponents ++= Seq("sql","hive", "mllib")

Sculpin