Where is the classpath set for hadoop

Question

Where is the classpath for hadoop set? When I run the below command it gives me the classpath. Where is the classpath set?

  bin/hadoop classpath

I'm using hadoop 2.6.0

Irrelevance

Answer 1

Open your bash profile (~/.profile or ~/.bash_profile) for editing and add the following:

1. export HADOOP_HOME="/usr/local/Cellar/hadoop" then Replace with your own path

2. export HADOOP_CLASSPATH=$(find $HADOOP_HOME -name '*.jar' | xargs echo | tr ' ' ':') Save the changes and reload.

3. source ~/.profile

Crossing

Answer 2

As said by almas shaikh it's set in hadoop-config.sh, but you could add more jars to it in hadoop-env.sh

Here is a relevant code from hadoop-env.sh which adds additional jars like capacity-scheduler and aws jar's.

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

# ... some other lines omitted

# Add Aws jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:share/hadoop/tools/lib/*

Rabato

Prefer for f in $HADOOP_HOME/share/hadoop/tools/lib/*.jar; do export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f done to just export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:share/hadoop/tools/lib/* – Mothball

What is the difference between simple export with wildcard and using for loop to add one by one..? – Elfie

Answer 3

When you run hadoop command, it sources a file hadoop-config.sh that resides in $HADOOP_HDFS_HOME/libexec which sets your classpath (CLASSPATH) by picking jars residing in various directories viz.

$HADOOP_HDFS_HOME/share/hadoop/mapreduce 
$HADOOP_HDFS_HOME/share/hadoop/common
$HADOOP_HDFS_HOME/share/hadoop/hdfs etc.

Salesgirl

In hadoop 2.6.0 HADOOP_HOME is changed to HADOOP_PREFIX – Irrelevance

Yeah either use that or use $HADOOP_HDFS_HOME. – Salesgirl

When I run bin/hdfs dfs -put etc/hadoop input I get the below error. put: `input': No such file or directory HOw can I resolve this error? – Irrelevance

So put syntax is `-put <local file location> <hdfs location>, i believe you are trying to say copy this file in directory "/input" i..e you are missing "/" root directory? – Salesgirl

Answer 4

As per this blog post, it is in an environment variable named HADOOP_CLASSPATH. You can set it as you would any other environment variable, the specifics of which depend on which shell you use. If you use bash, then you can call like export HADOOP_CLASSPATH=/path/to/wherever:/path/to/wherever/else.

Helminthic

Thanks. It looks like the blog is not active anymore! – Cricket

Answer 5

I also encountered the problem and have solved it, but my hadoop version is 2.10.1.

I hope it have some help for people who use a newer hadoop version. So far, the following methods should have worked as well in the latest hadoop version 3.3.0.

You just need to edit your .bashrc or .profile, I will give an example of .bashrc.

# edit .bashrc
$ vim ~/.bashrc

Add HADOOP_HOME, PATH of hadoop bin direcotry and HADOOP_CLASSPATH in .bashrc.

# export HADOOP_HOME=${your hadoop install directory}, an example as follows:

export HADOOP_HOME=/usr/local/hadoop-2.10.1

export PATH=${HADOOP_HOME}/bin:${PATH}

export HADOOP_CLASSPATH=`hadoop classpath`

Then,

$ source ~/.bashrc

Brookes

Answer 6

If you want to add classpath entries without changes to the installed files (eg, due to permission restrictions), you can use put a .hadooprc file in your home directory with following content:

hadoop_add_classpath /path/to/file1.jar
hadoop_add_classpath /path/to/file2.jar

The .hadooprc file gets sourced automatically by hadoop-config.sh if it exists.

Note this only adds classpath entries for the current user.

Bennion