how to run a class from java jar file on Hadoop?

Question

I have a jar file exported from Eclipse (Runnable JAR>>Copy required libraries into a sub folder).

In java if you set the main class in the manifest.xml you can run:

java -jar MyTest.jar arguments

if you want to run another main class in the jar file or if you didn't set a main class in the manifest.xml you can run:

java -cp MyTest.jar MyOtherMainClass arguments

In Hadoop if main class is set in manifest.xml you can run:

hadoop jar MyTest.jar arguments

If you type:

hadoop jar MyTest.jar MyOtherMainClass arguments

It will consider MyOtherMainClass as argument (not as a class to run) in the "args" array of the original main class of jar.

Now if you want to run another main class in the jar file what will you type?

I expect something similar to:

hadoop java -cp MyTest.jar MyOtherMainClass arguments

but that gives:

Error: Could not find or load main class java

Notice: if I remove "hadoop" from "hadoop java -cp MyTest.jar MyOtherMainClass arguments" it will launch normally

Chimb

Answer 1

The problem comes from Eclipse forcing you to set the main class in the jar file and hence preventing you to run the class that you want. All you have to do is remove the main class from the manifest.xml file of the jar file and run:

hadoop jar MyTest.jar MyOtherMainClass arguments

Take a look here: http://www.roman10.net/2012/07/26/specify-the-main-class-in-a-jar-file-for-hadoop/

I typed the same text in case the url deleted:

Hadoop support execution of jar file. For an executable jar file in normal java execution, one can specify the main class in the command line, as covered in my previous post: switch between main classes in a jar file.

However, the rules are a bit different for executable jar file running with hadoop. Basically the following rules hold (I tested on Hadoop 1.0.3),

If a jar file contains a main class specified in its manifest file, hadoop will take the main class even if the command specify another main class. This is different from normal java execution where we can specify a main class to overwrite the one in the manifest file. If a jar file does not contain a main class in manifest file, hadoop allows us to specify the main class. At eclipse, when one export a project as runnable jar file, it always ask for a main class at Launch configuration.

The main class selected will be put in the manifest file. Below is the content of the META-INF/MANIFEST.MF file in my helloworld project where the main class is set to HelloWorld.

Manifest-Version: 1.0 Class-Path: . Main-Class: hello.HelloWorld One can browse the jar file using a file extractor, open the manifest file using a file editor, and simply delete the last line to remove the main class configuration, and save the changes to the jar file when prompted. This will create a runnable jar file without main class.

The modified jar file can then be used in Hadoop with user supplied main class configuration, as shown in the sample command below,

$ hadoop jar hello.jar hello.HelloWorld

Chimb

Hi, I have a very silly problem that I got stock into. I have a simple(just printing hello world) .class file locally which I can convert it to .jar and run it locally with no problem. However, when I put it in my hadoop cluster I face with a silly bug: JAR does not exist or is not a normal file: /usr/reihan/test.jar I also should mention that I can run hadoop-builtin-examples like wordcount with no problem but when I want to run my own jar I face with this problem. any idea? – Lists

First, check if you are generating the jar file properly, an advice is to use eclipse to generate it for you. Second, try to use the full path of your jar file, and try before to see if it is already put in HDFS. Third, try to set the main class in your manifest file if it is not there. Last, if it didn't work write in a comment the command you are using and a link to your jar file so that I can take a look. – Chimb