how can I compile/using mahout for hadoop 2.0?

Question

The latest release mahout 0.9 is only built on hadoop 1.x. (mvn clean install) How can I compile mahout for hadoop 2.0.x?

Because When I was running the commands:

hadoop jar mahout-examples-0.9-SNAPSHOT-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -s SIMILARITY_COOCCURENCE -i test -o result

I always got the error message

IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.

Thanks!

Ballinger

Answer 1

To compile Mahout to work with 2.x since it isn't released in a package that is compatible with Hadoop 2.x:

mvn clean install -Dhadoop2 -Dhadoop2.version=2.2.0 -DskipTests=true

If you want to confirm after the build that the correct dependencies were brought in, run the following from the root of the project:

find . -name hadoop*.jar

The artifacts generated by the above command differ from what's in the 0.9 release so you'll need to update more than just the version number (there seems to be a new mahout-mrlegacy jar).

That being said, if someone can point me to an official artifact that runs with 2.x (even if it's a SNAPSHOT), I'd appreciate it.

Hardback

How smart and necessary is it to use Mahout with Hadoop 2.x? Thanks – Yamen

Answer 2

Did you try changing the version of hadoop in the pom.xml?

<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>                                                   
<version> 0.23.9</version> 

(Please check the version of hadoop. As I understand, 0.23 became 2.0)

Another thing to try is to run the RecommenderJob directly like this:

bin/mahout recommenditembased \
   --input INPUT --output OUTPUT \
   --tempDir TEMP --similarityClassname SIMILARITY_LOGLIKELIHOOD

To run this you have to be in the root of your mahout folder, and you need to have the following environment variables set:

export HADOOP_HOME=/path/to/hadoop/home
export HADOOP_COMMONS_HOME=/path/to/commons/home
export JAVA_HOME=/path/to/java/home
export PATH="$PATH:$HADOOP_HOME/bin"

Longawa

Thank you! <br/> I searched the version information of hadoop on maven website, but I found the version didn't have 0.23.9 or 2.0.0. search.maven.org/#browse%7C-1554017732 I need carry the experiment on clusters to test the cluster performance, so the non-distributed recommending system may not be fit for me. – Ballinger

Did you try my second suggestion, running mahout from your machine not as a hadoop jar? I always run it like this, and I had no problems. – Longawa

Answer 3

About an hour ago Mahout has officially added support to Hadoop 2.x on the master branch (see MAHOUT-1329)

Checkout the code here https://github.com/apache/mahout and recompile using:

mvn clean package -Dhadoop2.version=2.2.0

Try and see if that works.

Choctaw