Cascading examples failed to compile?

Question

In shell I typed gradle cleanJar in the Impatient/part1 directory. The output is below. The error is "class file for org.apache.hadoop.mapred.JobConf not found". Why did it fail to compile?

:clean UP-TO-DATE
:compileJava
Download http://conjars.org/repo/cascading/cascading-core/2.0.1/cascading-core-2.0.1.pom
Download http://conjars.org/repo/cascading/cascading-hadoop/2.0.1/cascading-hadoop-2.0.1.pom
Download http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.pom
Download http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.pom
Download http://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
Download http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.pom
Download http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.pom
Download http://repo1.maven.org/maven2/org/codehaus/janino/janino/2.5.16/janino-2.5.16.pom
Download http://conjars.org/repo/cascading/cascading-core/2.0.1/cascading-core-2.0.1.jar
Download http://conjars.org/repo/cascading/cascading-hadoop/2.0.1/cascading-hadoop-2.0.1.jar
Download http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar
Download http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar
Download http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar
Download http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
Download http://repo1.maven.org/maven2/org/codehaus/janino/janino/2.5.16/janino-2.5.16.jar
/home/is_admin/lab/cascading/Impatient/part1/src/main/java/impatient/Main.java:50: error: cannot access JobConf
    Tap inTap = new Hfs( new TextDelimited( true, "\t" ), inPath );
                ^
  class file for org.apache.hadoop.mapred.JobConf not found
1 error
:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 22.599 secs

Rodenhouse

I don't think this should be closed: The problem here is that gradle doesn't play nicely with java 7, and anyone going through the cascading tutorial with 7.x JDK will have this issue. The answer, which I'm happy to add, is to use a Java 6 JDK with the cascading tutorial. – Blockade

See also groups.google.com/d/topic/cascading-user/XEZapg4uqdI/discussion – Mesosphere

I believe this is because you don't have hadoop installed, and wordcount/build.gradle has "{ transitive = false }" so it doesn't install it. Remove that, and it'll build. – Mantel

I think that class got missed in the jar. please check the jar in your repository. – Candi

I think hadoop-conf jar is not present in classpath.Let me know in which hadoop version you are running this program. – Lightweight

If you are on OSX, you can download Java 6 here - developer.apple.com/downloads. Search for Java 6, "Java for OS X 2013-005 Developer Package" works. – Phototonus

Answer 1

try using Java 6 , gradle has some known issues with java 7

Aggie

Answer 2

Add hadoop-core-1.x.y.jar from hadoop distribution to path.

This worked for me in CoPA example. I added to eclipse build path.

Add this to gradle dependencies{} section

compile( group: 'org.apache.hadoop', name: 'hadoop-core', version: '1.2.0' )

Microreader

Answer 3

It seems like class not found, please check libraries in build path entries.

Hardball

Answer 4

I had the same issue today trying the Cascading for the Impatient tutorial. I have solved adding the following to my build.gradle in the dependenciessection :

compile 'org.apache.hadoop:hadoop-client:2.3.0'

I wanted to use Hadoop 2.3.0, but I guess that you can choose your own version of Hadoop.

PS : In order to use the first hadoop-mapred package, you have to add the following instead :

compile 'org.apache.hadoop:hadoop-mapred:0.22.0'

You can find all the repos you might need for Hadoop here.

Permeate

Answer 5

Thanks Lalit, Adding the below line did the trick for me without have to change any java version..

compile( group: 'org.apache.hadoop', name: 'hadoop-core', version: '1.2.0' )

Alternatively, I can even add the hadoop-core jar in the class path and mention that in the gradle path

Penetralia