How to import org.apache Java dependencies w/ or w/o Maven

Question

So the quick background is I am creating a java program, that uses many different imports

    import org.apache.hadoop.conf.*;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.*;
    import org.apache.hadoop.mapreduce.*;
    import org.apache.hadoop.util.*;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

I know that Netbeans isn't finding these files because I do not have them on my computer. But is there a way to have Netbeans automatically connect with org.apache and retrieve these files? Or do I just have to go and download them. Someone recommended using Maven, but I am not sure if this is the right solution or how to go about that?

Thanks

Anglicanism

Download the Hadoop JARs and put them in your project's /lib directory. You can get them here: hadoop.apache.org/common/docs/r0.20.2/quickstart.html. Why on earth would you make such a simple thing so complicated? – Bellamy

50% of this question was about how to do it this time, but the other 50% of the question was for how to do it the "right" way. – Anglicanism

So how is doing something simple "wrong"? – Bellamy

because if they update org.apache.hadoop I rather not need to worry about having to manual update the the program(which I might not have access to later). That's why I was asking about Maven – Anglicanism

And if they update the JARs in such a way that they break your code you won't have to worry about it until you're ready to accept them if you download them manually. Believe me, I know and understand the arguments. I don't agree that they're beneficial. – Bellamy

good point, thanks a bunch for the advice! – Anglicanism

Answer 1

Unless you use a Maven structure (see here getting started with Maven) you will have to download all jars manually.

If using only Hadoop (as in your example) this might not seem that much of a deal, but when working with big projects it is easier to declare your dependencies in a pom.xml file. It is much more easier than downloading X different jars, and you can easily move to a newer version of a library, rather than having to delete and and download another.

I saw that someone asked in a comment why people like Maven so much. Well, to be honest, I personally find it easy to use and very useful. Furthermore, a Maven project can be easily imported in IntelliJ, Eclipse or Netbeans, whereas creating for example an IntelliJ project can cause difficulties in importing it in Eclipse or NetBeans.

To get started using Maven with Netbeans, you can go to: New Project, Categories:Maven Projects:{Best Option}. Then in the project files, open pom.xml. Here is where dependencies for your project are added. If you are not sure what to insert try searching for your jar name + "maven" on the internet. The plugin for Netbeans is able to connect to the maven repository and autocomplete most fields.

Sample from: http://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.6

<project...>
....
<dependencies>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.0.3</version>
    </dependency>
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.6</version>
    </dependency>
</dependencies>
...
</project>

Maestoso

For a moment I asked myself why you were keeping your dependencies in a porn.xml file – Indulgence

Answer 2

Download the .jar file here: http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core/0.20.2

In Eclipse, right-click on your project, click Properties, search in the text box for Library, click on Build Paths, click Add External JAR, and select the file you downloaded from the link above.

Combustion

Answer 3

You will have to download the jar-files yourself. Unless you start using Maven or a similar dependency management tool.

Odette

Answer 4

You must download them. The name org.apache.hadoop is a package name, and we only use the name of the site as a convention. See this tutorial on packages for more information. Essentially a package is a folder on your computer, often in the Java\jre\lib\ext\ directory.

Indulgence

Answer 5

Refer tutorial http://www.tutorialspoint.com/hadoop/hadoop_mapreduce.htm

It mentions :-

Download Hadoop-core-1.2.1.jar, which is used to compile and execute the MapReduce program. Visit the following link http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core/1.2.1 to download the jar.

Ululate

Answer 6

Not a netbeans user , but I'm sure that even in netbeans, you have a maven plugin. "Mavenize" your project, and when you will perform mvn clean install, you will get these jars to local maven repository.
With Eclipse I use the m2Eclipse plugin and it works really well for me.
This of course depends that these jars can be found in maven repositories over the net, such as maven central repository.

Monney

I do not understand why people like Maven so much. – Bellamy

I do not understand why people dislike Maven so much. :-) – Maremma

People? No, just me. It's the unnecessary complexity for insufficient benefit that kills me. Plus the mandates on directory structure that as far as I know cannot be altered. – Bellamy

You can use ivy instead of maven. I can tell you that with Eclipse you can generate a maven project quite easy. – Monney

I can generate with Maven and IntelliJ just fine. That's not my point. I see too many people coming here because they're stuck when the root cause is larding too many tools on top of a poor understanding. Better to learn without all the "conveniences". And I greatly prefer Ivy and Ant to Maven. I hate being forced to use their conventions. – Bellamy

Answer 7

I have final figured out my preferred way to create a new Hadoop project and import the dependencies using Maven.

Using NetBeans I create a new Maven project.

Then under project files, I open the pom.xml.

I finally add inside of

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.2</version>
    </dependency> 

After building with dependencies I am now ready to code.

Anglicanism