How to add Spark to Maven project in Eclipse?

Question

I would like to start Spark project in Eclipse using Maven. I've installed m2eclipse and I have a working HelloWorld Java application in my Maven project.

I would like to use Spark framework and I'm following directions from the official site. I've added Spark repository to my pom.xml:

<repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
</repository>

And then the dependency:

<dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>0.9.9.4-SNAPSHOT</version>
</dependency>

But I'm getting an error in Eclipse:

Missing artifact spark:spark:jar:0.9.9.4-SNAPSHOT

How can I resolve this issue? I don't want to download Spark's jar file and place in the local repository.

This is my pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.myproject</groupId>
  <artifactId>Spark1</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>Spark1</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <repository>
      <id>Spark repository</id>
      <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
  </repository>

  <dependencies>
<!--     (...) -->

    <dependency>
      <groupId>spark</groupId>
      <artifactId>spark</artifactId>
      <version>0.9.9.4-SNAPSHOT</version>
    </dependency>

  </dependencies>

</project>

Et

Could you show us your whole pom.xml? You can omitt other dependencies but I'd like to see the structure. Also make sure your Eclipse project really has Maven enabled (rigt click on project=>Configure=>Convert to Maven Project). – Avionics

@André I've added pom.xml file to description. – Et

Answer 1

The repository block needs to be wrapped in a repositories block:

<repositories>
    <repository>
        <id>Spark repository</id>
        <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
    </repository>
</repositories>

Unbodied

Answer 2

Currently no repository is required to add for Spark library loading

You just need to add

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-core</artifactId>
    <version>2.6.0</version>
</dependency>

And that's it.

Useful tutorials to play with is here

Endocardium

Answer 3

Reason for failure is 0.9.9.4-SNAPSHOT is not available.Below is the list of the snapshots available. Use one among them based on your requirement.

0.9.8-SNAPSHOT/ Sat May 21 21:54:23 UTC 2011
0.9.9-SNAPSHOT/ Mon May 23 10:57:38 UTC 2011
0.9.9.1-SNAPSHOT/ Thu May 26 09:47:03 UTC 2011
0.9.9.3-SNAPSHOT/ Thu Sep 01 07:53:59 UTC 2011

Thanks, Sankara Reddy

Boxer

Answer 4

I had run into same issue because initially I started with different repository url for spark and then to use earlier version I changed the repository url. Some how it didn't seem to come into effect until I changed the repository id. Try changing the repository id.
Could be bug in maven because running maven from console also couldn't resolve the dependency without updating the id.

Outlier

This actually worked for me.. all I did was I changed id from "Spark repository" to "Spark Repository". – Salba

Answer 5

please add the repository , tag inside the repositories tag like below

<repositories>
        <repository>
            <id>Spark repository</id>
            <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url>
        </repository>
    </repositories>

Palsy

Answer 6

The last versions (2.1 and later) of Spark only need the dependency defined inside the pom.xml file

<dependency>
    <groupId>com.sparkjava</groupId>
    <artifactId>spark-core</artifactId>
    <version>2.1</version>
</dependency>

the repository definition is not longer required

Hymenopterous

Answer 7

use this latest repository. http://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10/1.6.0

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>1.6.0</version>
</dependency>

Axiology

Answer 8

use this, and also make sure you change spark library to version 2.11.x in eclipse project build path

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.spark-scala</groupId>
    <artifactId>spark-scala</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>${project.artifactId}</name>
    <description>Spark in Scala</description>
    <inceptionYear>2010</inceptionYear>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.tools.version>2.10</scala.tools.version>
        <!-- Put the Scala version of the cluster -->
        <scala.version>2.10.4</scala.version>
    </properties>

    <!-- repository to add org.apache.spark -->
    <repositories>
        <repository>
            <id>cloudera-repo-releases</id>
            <url>https://repository.cloudera.com/artifactory/repo/</url>
        </repository>
    </repositories>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <!-- see http://davidb.github.com/scala-maven-plugin -->
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.13</version>
                <configuration>
                    <useFile>false</useFile>
                    <disableXmlReport>true</disableXmlReport>
                    <includes>
                        <include>**/*Test.*</include>
                        <include>**/*Suite.*</include>
                    </includes>
                </configuration>
            </plugin>

            <!-- "package" command plugin -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>
</project>

Meadow