Where HDFS stores files locally by default?

Question

I am running hadoop with default configuration with one-node cluster, and would like to find where HDFS stores files locally.

Any ideas?

Thanks.

Accused

Answer 1

You need to look in your hdfs-default.xml configuration file for the dfs.data.dir setting. The default setting is: ${hadoop.tmp.dir}/dfs/data and note that the ${hadoop.tmp.dir} is actually in core-default.xml described here.

The configuration options are described here. The description for this setting is:

Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.

Raseda

this link is broken :( – Baggy

what is the default value of hadoop.tmp.dir ? – Sunn

From the above core-default.xml link, it shows hadoop.tmpdir = /tmp/hadoop-${user.name} – Farmhouse

Answer 2

Seems like for the current version(2.7.1) the dir is

/tmp/hadoop-${user.name}/dfs/data

Based on dfs.datanode.data.dir, hadoop.tmp.dir setting from: http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/core-default.xml

Bondman

Answer 3

As "more recent answer" and to clarify hadoop version numbers:

If you use Hadoop 1.2.1 (or something similar), @Binary Nerd's answer is still true.

But if you use Hadoop 2.1.0-beta (or something similar), you should read the configuration documentation here and the option you want to set is: dfs.datanode.data.dir

Fusiform

Answer 4

For hadoop 3.0.0, the hdfs root path is as given by the property "dfs.datanode.data.dir"

Cannon

Answer 5

First find the Hadoop directory present in /usr/lib. There you can find the etc/hadoop directory, where all the configuration files are present.

In that directory you can find the hdfs-site.xml file which contains all the details about HDFS. There you find 2 properties:

dfs.namenode.name.dir – tells where the namenode stores the metadata on the local filesystem.

dfs.datanode.data.dir – tells where the datanode stores the data on the local filesystem

Acculturation

Answer 6

Run this in the cmd prompt, and you will get the HDFS location:

bin/hadoop fs -ls /

Sconce

As presented, this command just gives an error. And you fixed the command, it wouldn't answer the original question at all, -ls isn't going to help in this regard. – Nylon