HDFS Home Directory

Question

I have setup a single node multi-user hadoop cluster. In my cluster, there is an admin user that is responsible for running the cluster (superuser). All other users are allocated a hdfs directory like /home/xyz where xyz is a username.

In unix, we can change the default home directory for a user in /etc/passwd. And by default, landing directory for a user is the home directory.

How do I do it in hadoop for hdfs file system. I want for example, if user types: $hadoop dfs -ls on the unix prompt. It shall list the contents of the home directory allocated by me.

Further, hdfs directories are created by the superuser who runs the cluster(hadoop superuser and not unix root) and then transfers the ownership to a particular user.

Houlihan

Answer 1

I'm not sure this is something that can be configured - the source for DistributedFileSystem(line 150) has a call for getHomeDirectory that seems to be hard-coded:

@Override
public Path getHomeDirectory() {
  return makeQualified(new Path("/user/" + dfs.ugi.getShortUserName()));
}

You do have two possible choices if you want to be able to change this:

• Submit a ticket to hadoop asking for a new feature - See this link
• Amend the source yourself and re-build + re-distribute the hadoop-core jar across your cluster (simple in your single node pseudo cluster)

Monicamonie

Hi Chris, cool observation! I will try to rebuild my source but was just wondering if I just need to make a single change as mentioned by you or there are some other functions where I may need to change it. – Houlihan

From a hadoop source dir, execute: grep "\"/user" * -iRn --include=\*.java - this will catch most of the files, but maybe not all of them. You may not have to amend all the various unit tests. Good luck! – Monicamonie

Answer 2

tapan, Each user has a "home" directory in HDFS located at /user/username. For example, If you type hadoop fs -ls, it will take you the current HDFS user directory. I checked it on single with two users.

Coming to unix like file (/etc/passwd) setting, i did not know. But, it is really interesting. As chris said, we can add new feature like, introduce user-admin-site.xml like configuration file and add groups, users their permissions on directories, home directory etc. Really intertesting, we have o try this...

Declivity