Why does "hadoop fs -mkdir" fail with Permission Denied?

Question

I am using Cloudera on a VM machine that I am playing around with. Unfortunately I am having issues copying data to the HDFS, I am getting the following:

[cloudera@localhost ~]$ hadoop fs -mkdir input
mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

I am not too concerned about security on this VM, is there anyway I can open up security more on HDFS?

Meatman

Answer 1

Using mkdir in hadoop needs the "hadoop file permissions". From your example you can see that hdfs is a user that has permissions to create folders. So if you run:

sudo -u hdfs hadoop fs -mkdir /import

then the import folder will be created. If you want to change the owner of this folder run:

sudo -u hdfs hadoop fs -chown new_user /import

Now the new_user can manipulate files inside the import folder

Hoang

Answer 2

When you execute the above command, if hdfs home directory(/user/cloudera) is not there then that directory will be created first then the directory input will be created under /user/cloudera

For giving permission for cloudera user to create it's own directory, you got to give permission. hdfs user is the admin user in hdfs switch to hdfs then execute the following command

[hdfs@localhost~]$ hadoop fs -mkdir /user/cloudera ; hadoop fs -chmod 777  /user/cloudera

Or

if you are not too concerned about hdfs security you disable hdfs permission by setting the below property to false in hdfs-site.xml

<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>

after setting this property to false hdfs needs to be restarted.

Alice

if i run that command I get: [cloudera@localhost ~]$ hadoop fs -mkdir /user/cloudera ; hadoop fs -chmod 777 /user/cloudera mkdir: Permission denied: user=cloudera, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x chmod: `/user/cloudera': No such file or directory – Meatman

Before executing the above command you got to switch the user to hdfs user.. Use the command su hdfs – Alice

Or disable hdfs permission by following the second step – Alice

Answer 3

In cloudera manager, you can change the settings: hdfs->configuration->view&edit, uncheck the Check HDFS Permissions dfs.permissions and restart the hdfs.

Rebekah

Answer 4

I resolved the issue by creating a supergroup in /etc/group and updated the user logins on it. I mean user should be part of HDFS supergroup to have access to write on HDFS.

$vi /etc/group

supergroup:x:30000:root

Later was able to write on HDFS. Hope it helps

Pushkin

Answer 5

This is because you dont have enough permission to create directory in hdfs. Try running this as sudo:

sudo -u hdfs hadoop fs -mkdir -p /user/samplefolder

But this is not recommended because it compromises security.

Towill

Answer 6

export HADOOP_USER_NAME=hdfs

Try running this command and then make a directory, it worked for me like a charm.

Bellamy

Answer 7

To solve this problem you need to start the Cloudera express server by the following command:

$sudo /home/cloudera/cloudera-manager --force --express

Afterward, go to the server and try the commands again.

This solved my problem, I hope it'll solve yours as well.

Lated

Answer 8

In a general case, if you have a 'permission denied' error while running mkdir in hadoop.

1. Check if the permissions of the parent directory hadoop fs -ls /user to see if it has write permissions. If it doesn't, change the permission with something like hadoop fs -chmod -R 775 /user.
2. If two users belong to the same group but cannot write to the directory (even with write permissions). Check the user groups with hdfs groups <username> to ensure hadoop recognizes the group.

In my case, I had to do the second one and change the group with chgrp.

Cuspidate