How can I list subdirectories recursively for HDFS?

Question

I have a set of directories created in HDFS recursively. How can list all the directories ? For a normal unix file system I can do that using the below command

find /path/ -type d -print

But I want to get the similar thing for HDFS.

Repetend

Useful question, but I would like to know the size of directories, to better replicate du functionality. – Papal

Answer 1

To list directory contents recursively hadoop dfs -lsr /dirname command can be used.

To filter only directories , you can grep "drwx" (since owner has rwx permission on directories) in output of above command.

Hence whole command will look like as below.

$hadoop dfs -lsr /sqoopO7 | grep drwx 

Acrophobia

Answer 2

The answer given by @Shubhangi Pardeshi is correct but for latest hadoop version command has deprecated. So new latest command can be used as below

hdfs dfs -ls -R /user | grep drwx

Unconstitutional

Answer 3

The following method should be more robust to only get directories because it depends less on the permissions.

hdfs dfs -ls -R /folder | grep "^d"

Reluctivity